Tips, tweaks & hacks – Data Science, Data Analytics and Machine Learning Consulting in Koblenz Germany https://www.rene-pickhardt.de Extract knowledge from your data and be ahead of your competition Tue, 17 Jul 2018 12:12:43 +0000 en-US hourly 1 https://wordpress.org/?v=4.9.6 Cleaning up my network connections on ubuntu linux using the network manager nmcli https://www.rene-pickhardt.de/cleaning-up-my-network-connections-on-ubuntu-linux-using-the-network-manager-nmcli/ https://www.rene-pickhardt.de/cleaning-up-my-network-connections-on-ubuntu-linux-using-the-network-manager-nmcli/#respond Thu, 16 Apr 2015 19:03:34 +0000 http://www.rene-pickhardt.de/?p=1955 After 4 years of running a pretty stable linux on my notebook I realized that the time had come that too much software and dependencies have been on my system so I set up a clean system. By doing so I also switched to a tiling window manager called awesome (with which I am pretty happy so far). One problem (not really since it is the purpose of going to awesome) though is that everything has to be done from the command line. in particular when I join a network I have to use the network manager command line interface nmcli to set up my wireless connection or my wired connection. That is not too much of a problem but since I did a backup of all my old network connections I had quite a list of connections too look for in order to find the uuid of the network I wanted to join and enable the network with the suitable command. So I wanted to delete all the connections from hotels, airports and places that I am not visiting anymore. Obviously I could have done this by hand. But its much more fun to do it automatically and on the long term it is really faster when you are mastering your bash tools (:
So here we go with a step by step explanation of the following command that will remove all your network connections that you have never used:

$ for i in `nmcli c | grep "never" | grep -o -- "[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{12\}"` ; do nmcli connection delete uuid $i ; done

ok let us first see the connections that I had in my manager

$ nmcli connection

which would give me the following list

NAME UUID TYPE TIMESTAMP-REAL
Lobby 0f91bb0d-e2be-4f8a-a00e-457c8bdaf9a9 802-11-wireless never
TTH-Zentral 2d3ddca9-772f-47d9-99a1-1559640b0f25 802-11-wireless never
attwifi 6be3c5aa-fc85-415c-96e4-5583b25c23bb 802-11-wireless never
uni-koblenz 8a0d1d51-672c-4ded-ad92-18e27b8215df 802-11-wireless Do 16 Apr 2015 18:11:45 CEST
greenscafe 22dd0f8e-be2d-4402-af82-7716222add75 802-11-wireless never
WLAN-2DD138 5308ed9e-5def-430e-85f4-e1eb01426927 802-11-wireless never
Fairfield_GUEST b72080e0-c5dd-492a-b8a3-017fe7d6099c 802-11-wireless never
Hotel Sylter Hof d7c8fc6c-64e3-435f-b2c0-316c1f11ddf1 802-11-wireless never
TELENETHOTSPOT 1345bcda-26ab-4193-904c-088235d29873 802-11-wireless never
CHI_ECRC_2013 a316f6be-bb4c-4501-867b-6928f9f429ef 802-11-wireless never
AndroidAP5270 b5c5462c-072e-4e90-a3b2-def467562579 802-11-wireless never
hotel_harvey 7c6d1794-6d45-40aa-b7ac-f9a7f401d02e 802-11-wireless never
free-hotspot.com dc7351ce-ca6c-413e-ae8b-a82c2093525e 802-11-wireless never
PAT-WLAN 4662b075-9519-4f69-8ce9-b938ea8270c1 802-11-wireless never
dlink 5dab3493-cdd8-4d6c-a7a2-6def0836c2a5 802-11-wireless never
TTHKasse b9c2f9d9-aeda-454b-941b-bb1ccfc0ad9a 802-11-wireless never
BTOpenzone-B bede9d3a-9fc1-4dc2-9c75-0d2686c225f1 802-11-wireless never
MBTA_WIFI_Car0385_Box-068 17ac549c-574c-4bf4-a464-6b633f2a4ac3 802-11-wireless never
Wireless connection 1 17b18b79-411d-4d69-813d-6506f38b8ea5 802-11-wireless never
MBTA_WiFi_Car0620_Box-199 c36c572e-e469-4e86-a50f-87d6d68f4f7e 802-11-wireless never
*VIPARIS_WIFI 0593b927-644b-4604-a479-4fc652b6050d 802-11-wireless never
test 836af61a-8268-4a16-a5ab-036f3e0cf7e9 802-11-wireless So 15 Sep 2013 21:18:35 CEST
WL1A 46b6496f-6583-4306-a58f-4e4d2bdd5d50 802-11-wireless never
Hochhaus f6b80571-37d5-4bcd-b558-62084c61622b 802-11-wireless never
FreeWifi 379f9efd-b91d-433f-8ce2-209e4ed2099b 802-11-wireless never
NETCONNECT-6202 2f84b475-a852-49ac-a236-61b3f0bc9548 802-11-wireless never
stolteraa d789f49e-2643-4894-96a8-4e10ea3cbb69 802-11-wireless never
ALICE-WLAN28 17e4838f-bbaf-470c-bf24-9de948b42b00 802-11-wireless never
NAS QNAP ef6a08df-1c7c-48ba-b476-f3ff3b1e2669 802-3-ethernet Mi 15 Apr 2015 11:03:51 CEST
WLAN-8CA902 3046307e-69a3-4c99-89fb-b79f7f9ee24b 802-11-wireless Mi 01 Apr 2015 11:42:27 CEST
MfN-Guest 25d3765f-dcca-4f40-872a-8ffcfe307f25 802-11-wireless never
FRITZ!Box Fon WLAN 7360 7c428f50-078d-442a-9ba7-268853c888f0 802-11-wireless Mo 11 Feb 2013 23:39:08 CET
Arcor-362007-L 87a2d11c-3a3d-4b5e-a1b0-cc5b38916035 802-11-wireless Di 14 Apr 2015 17:42:33 CEST
Telekom_ICE 8763adcc-f27b-441f-9184-594128871351 802-11-wireless So 29 Mär 2015 19:56:35 CEST
IBMVISITOR 48227bb6-63e8-4a58-8b8f-1929f76d8ef2 802-11-wireless never
OWL 13ff2ed5-8ecf-4d67-b039-a3bc4cce56d9 802-11-wireless never
Parkhotel Kraehennest e763a256-4a2d-4813-b354-57ad8f642e2a 802-11-wireless never
EasyBox-AD3112 e4f342ae-d20b-4abc-9fe2-52b208d6b61b 802-11-wireless never
BTOpenzone 860cf4c0-45eb-4a57-a01e-cfd54933387f 802-11-wireless never
southshore 274cfaa8-2c08-4abc-bef3-3fd2608060b8 802-11-wireless never
annanet 06b3b3bc-a264-4d05-a848-3b5d6df9eed0 802-11-wireless never
shared 806c9806-fadc-472f-8460-318babbd38ea 802-3-ethernet Fr 10 Apr 2015 09:39:50 CEST
vidiu 3abf0848-b09c-457e-9c7a-958728f2c59f 802-11-wireless never
wlan 1 ad9e4a07-b66f-4f3f-8f6d-4f5a81b2ed54 802-11-wireless never
CITYHOTEL dec05243-06b6-4477-8813-0491b436f993 802-11-wireless never
eduroam f844713b-a726-4b5b-98a5-3b35411c4cf5 802-11-wireless never
Starlight-HotSpot 6842159f-94e9-4f5d-8a76-93b1f4c32912 802-11-wireless never
Dr_l)P35_21_22#342C5C e307b6e8-598b-4bd1-b47f-dc8a34ff12e0 802-11-wireless never
iPhone 5 9070ee1c-8ac1-4458-a590-cf241685b929 802-11-wireless never
Lummerland 7d2d4d89-9eb7-45c0-a963-a8a240e33100 802-11-wireless Mo 10 Feb 2014 22:35:03 CET
AndroidAP4089 ec201acc-9e6f-4c7b-9da8-6ed4539dde6c 802-11-wireless never
heartofgold 437e2b7d-5b76-4e3e-a101-698776293ede 802-11-wireless never
09FX09039648 e2cc4508-e18e-4c0c-a694-633ace5d095a 802-11-wireless never
WIRELESS_BEN_GURION_AIRPORT 8ff2846d-9e84-4175-8c4d-968e226e7a43 802-11-wireless never
Telekom 60aded97-f697-48c6-8070-91e1b786df82 802-11-wireless Mi 15 Apr 2015 00:14:14 CEST
Marriott_CONF a4ec2c22-8a31-46bb-a146-a02d8f7b2582 802-11-wireless never
TTH-Seminar4+5 23422256-0a4b-45b5-8394-43c92592564d 802-11-wireless never
FRITZ!Box 7330 SL a431f83e-6a73-4cf1-94ee-4409c19dfdb9 802-11-wireless Do 13 Feb 2014 18:45:48 CET
mycloud f3fcf1ae-eebb-4e37-a700-96035d0aea59 802-11-wireless never
ArcorWirelessLAN 428769c9-3dfc-43f0-a2dc-29f3c4ecb0fb 802-11-wireless never
TP-LINK_PocketAP_7329E2 a17a19b4-251e-41d7-a82e-1add52ec1317 802-11-wireless never
TTH-Taunus fbb24cae-c2aa-4057-be67-095b80604f5e 802-11-wireless never
BWI-WiFi 58e749f6-58e9-44fe-8528-6a1e3bacc88f 802-11-wireless never
uni vpn b28d76f8-57d2-4548-8959-9c47e189221e vpn never
Miri 🙂 74147f40-07c4-4e91-8a43-8c69bc733479 802-11-wireless never
wiew7 7193e045-9570-4d16-b4b6-d676fc19d05a 802-11-wireless never
ZurichAirport 0fe8cde1-de8b-48f9-9ca0-52ac129e9876 802-11-wireless never
mercure f3fbf40d-7c2b-4e6d-b5db-fce6888bb719 802-11-wireless never
HITRON-A600 501ac6d9-6912-46a1-9681-4a73e1009fe0 802-11-wireless never
HTC Portable Hotspot 61EF 3675cadc-251a-487c-8c4d-cf3747117f13 802-11-wireless never
FRITZ!Box 7312 a7d9b99d-63bc-4caa-b82c-1da9fec95cea 802-11-wireless never
o2-WLAN38 76a67dd7-3d50-4388-ba9f-07f52a328eca 802-11-wireless never
ALICE-WLAN36 e55d0ccd-7941-4da0-8ac8-99ed22adda49 802-11-wireless never
WEBPORTAL e2ce59eb-fa7d-4ece-88b7-bd0585a8d589 802-11-wireless never
FOSDEM 0cc3584b-1a5e-41fd-944e-df85f0a2d418 802-11-wireless never
BRITZ!Fox Fon WLAN 7360 SL fc70206c-04c8-4ae6-a0fd-7f769addf0b6 802-11-wireless never
Hotel Amsee3 f4ff9826-22d2-4f9b-91d9-8c63dc0f9bd4 802-11-wireless So 29 Mär 2015 11:56:21 CEST
ibis a348ed30-2fcf-4068-ba1c-176cdd9e7b10 802-11-wireless never
O2 Wifi 8935490d-313f-4fd0-b1fd-c78ab138af51 802-11-wireless never
FRITZ!Box Fon WLAN 7270 50d62587-5a48-4635-b30f-fbb1ded20455 802-11-wireless Di 30 Dez 2014 12:23:23 CET
Ambassador-Opera fdefea8c-d218-4632-bb07-b67ddb63f778 802-11-wireless never
HOTEL BB 05d9792c-6f9e-48a6-9c0c-6c03a9708954 802-11-wireless never
MBTA_WiFi_Car0353_Box-104 24a1527c-3083-44f3-8812-cacafb88cf7a 802-11-wireless never
theairline 54770eb9-78f2-4a3b-86cf-0e594a55c9f9 802-11-wireless never
republicansareontheirown 074ca340-643c-411e-95eb-d6e78f887fcf 802-11-wireless never
VPN/WEB 4a5f132b-81c2-49ba-8934-0257f98e5a54 802-11-wireless never
WLAN-6A1EA7 d2e80ec0-9bd3-4801-bada-65a695f7dc92 802-11-wireless never
Mahler 1d59c9a1-090c-4333-b2af-4154d5edd9c2 802-11-wireless never
guest-access c0ff03ac-a195-46a7-a507-5c1cf5c7f668 802-11-wireless never
Schlummerland 3d029511-2627-458a-a06b-e2c8e86d26b4 802-11-wireless never
MSDSL2 f2fbd0b7-26f6-4939-aece-c2c9d3621e40 802-11-wireless Do 16 Apr 2015 20:28:22 CEST
Urania 846e76bc-b96f-40aa-a30f-b6d2f84fd7f0 802-11-wireless never
CJDDSAWLN 55158cdb-07cc-4c02-8f44-a19ec2aeca06 802-11-wireless never
AndroidAP 375e398a-1b2e-4d2e-b4a5-4c05c904b109 802-11-wireless Fr 02 Jan 2015 16:29:17 CET
gesis-guest b9a00edd-9e28-4549-8f13-2634577c1276 802-11-wireless never
guests@WMDE 9aeccb7b-e629-4f0a-ab81-5300c7c433c0 802-11-wireless never
Wired connection 1 30ffcf0e-5181-41b0-b7d4-402de875889f 802-3-ethernet Mo 04 Mär 2013 15:10:44 CET
wlan a2f073a8-f0b0-4f58-a79a-2f22131016b1 802-11-wireless never
Wired connection 2 700ebb58-f049-4ec7-bac6-cd5801a975e2 802-3-ethernet Do 16 Apr 2015 18:11:44 CEST
Wired connection 3 bf70a943-9510-4883-ba64-a4bede826120 802-3-ethernet Do 16 Apr 2015 09:26:08 CEST
30 Min. Free Wi-Fi - Vodafone f5cd9efa-d735-41d5-a90f-ba83357f2e40 802-11-wireless never
WIKIMEDIA a40d0c35-8232-42db-b5a2-2118d69d6c41 802-11-wireless never
loganwifi 41e42d9c-efe9-45c5-9f8e-a2f5f441dd73 802-11-wireless never
WL3D efe99726-557a-4595-90e8-9c0fc3ee0c20 802-11-wireless never
m3connect 6ce7f634-5c44-4818-a055-cb32dca20738 802-11-wireless never
MIRI-PC_Network d1243377-9aed-4abf-8207-90b47840a48f 802-11-wireless never
innflux bd5be404-6a8b-402a-8a6f-856eaa054664 802-11-wireless never
LRTA24open2 dc95a098-107f-46c9-8d36-23cf575a319f 802-11-wireless never
LRTA24open3 5166cb9a-20ff-4d5c-b224-88f0b2276398 802-11-wireless never
Boingo Hotspot 954559e9-5f8c-4f9b-b2bc-36ff23f18d4a 802-11-wireless never

the interesting part is the fourth column which is luckily saying never if a connection was never used. so that is an easy grep:

nmcli c | grep "never"

from this list I need the second colum in particular the uuids. I could have done this with some awk magic but I decided greping for uuids should be easier so lets pipe the output to another grep:

nmcli c | grep "never" | grep -o -- "[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{12\}"

as we can see the regular expression is really straight forward. there are two arguments given first of all -o that is to only output what was matched and not the complete lines and the helps for greping hyphens. The cool thing is with having the web and search engines you don’t even have to build this regular expression on your own. Click on my search query for an blog article explaining the background: grep regular expression uuid
Next we have to iterate over the result and use it in the nwcli connection delete command. From the docu of nmcli we know

Usage: nmcli connection { COMMAND | help }
COMMAND := { list | status | up | down | delete }
list [id | uuid ]
status [id | uuid | path ]
up id | uuid [iface ] [ap ] [--nowait] [--timeout ]
down id | uuid
delete id | uuid

so the code should look something like:

nmcli connection delete uuid 954559e9-5f8c-4f9b-b2bc-36ff23f18d4a

only that 954559e9-5f8c-4f9b-b2bc-36ff23f18d4a will be replaced by all the uuids from my grep statements.
so let’s put this big grep in a loop and just echo everything to see if it is working. just use backticks around the grep and make a loop:

$ for i in `nmcli c | grep "never" | grep -o -- "[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{12\}"` ; do echo $i ; done
0f91bb0d-e2be-4f8a-a00e-457c8bdaf9a9
2d3ddca9-772f-47d9-99a1-1559640b0f25
6be3c5aa-fc85-415c-96e4-5583b25c23bb
22dd0f8e-be2d-4402-af82-7716222add75
5308ed9e-5def-430e-85f4-e1eb01426927
b72080e0-c5dd-492a-b8a3-017fe7d6099c
d7c8fc6c-64e3-435f-b2c0-316c1f11ddf1
1345bcda-26ab-4193-904c-088235d29873
a316f6be-bb4c-4501-867b-6928f9f429ef
b5c5462c-072e-4e90-a3b2-def467562579
7c6d1794-6d45-40aa-b7ac-f9a7f401d02e
dc7351ce-ca6c-413e-ae8b-a82c2093525e
4662b075-9519-4f69-8ce9-b938ea8270c1
5dab3493-cdd8-4d6c-a7a2-6def0836c2a5
b9c2f9d9-aeda-454b-941b-bb1ccfc0ad9a
bede9d3a-9fc1-4dc2-9c75-0d2686c225f1
17ac549c-574c-4bf4-a464-6b633f2a4ac3
17b18b79-411d-4d69-813d-6506f38b8ea5
c36c572e-e469-4e86-a50f-87d6d68f4f7e
0593b927-644b-4604-a479-4fc652b6050d
46b6496f-6583-4306-a58f-4e4d2bdd5d50
f6b80571-37d5-4bcd-b558-62084c61622b
379f9efd-b91d-433f-8ce2-209e4ed2099b
2f84b475-a852-49ac-a236-61b3f0bc9548
d789f49e-2643-4894-96a8-4e10ea3cbb69
17e4838f-bbaf-470c-bf24-9de948b42b00
25d3765f-dcca-4f40-872a-8ffcfe307f25
48227bb6-63e8-4a58-8b8f-1929f76d8ef2
13ff2ed5-8ecf-4d67-b039-a3bc4cce56d9
e763a256-4a2d-4813-b354-57ad8f642e2a
e4f342ae-d20b-4abc-9fe2-52b208d6b61b
860cf4c0-45eb-4a57-a01e-cfd54933387f
274cfaa8-2c08-4abc-bef3-3fd2608060b8
06b3b3bc-a264-4d05-a848-3b5d6df9eed0
3abf0848-b09c-457e-9c7a-958728f2c59f
ad9e4a07-b66f-4f3f-8f6d-4f5a81b2ed54
dec05243-06b6-4477-8813-0491b436f993
f844713b-a726-4b5b-98a5-3b35411c4cf5
6842159f-94e9-4f5d-8a76-93b1f4c32912
e307b6e8-598b-4bd1-b47f-dc8a34ff12e0
9070ee1c-8ac1-4458-a590-cf241685b929
ec201acc-9e6f-4c7b-9da8-6ed4539dde6c
437e2b7d-5b76-4e3e-a101-698776293ede
e2cc4508-e18e-4c0c-a694-633ace5d095a
8ff2846d-9e84-4175-8c4d-968e226e7a43
a4ec2c22-8a31-46bb-a146-a02d8f7b2582
23422256-0a4b-45b5-8394-43c92592564d
f3fcf1ae-eebb-4e37-a700-96035d0aea59
428769c9-3dfc-43f0-a2dc-29f3c4ecb0fb
a17a19b4-251e-41d7-a82e-1add52ec1317
fbb24cae-c2aa-4057-be67-095b80604f5e
58e749f6-58e9-44fe-8528-6a1e3bacc88f
b28d76f8-57d2-4548-8959-9c47e189221e
74147f40-07c4-4e91-8a43-8c69bc733479
7193e045-9570-4d16-b4b6-d676fc19d05a
0fe8cde1-de8b-48f9-9ca0-52ac129e9876
f3fbf40d-7c2b-4e6d-b5db-fce6888bb719
501ac6d9-6912-46a1-9681-4a73e1009fe0
3675cadc-251a-487c-8c4d-cf3747117f13
a7d9b99d-63bc-4caa-b82c-1da9fec95cea
76a67dd7-3d50-4388-ba9f-07f52a328eca
e55d0ccd-7941-4da0-8ac8-99ed22adda49
e2ce59eb-fa7d-4ece-88b7-bd0585a8d589
0cc3584b-1a5e-41fd-944e-df85f0a2d418
fc70206c-04c8-4ae6-a0fd-7f769addf0b6
a348ed30-2fcf-4068-ba1c-176cdd9e7b10
8935490d-313f-4fd0-b1fd-c78ab138af51
fdefea8c-d218-4632-bb07-b67ddb63f778
05d9792c-6f9e-48a6-9c0c-6c03a9708954
24a1527c-3083-44f3-8812-cacafb88cf7a
54770eb9-78f2-4a3b-86cf-0e594a55c9f9
074ca340-643c-411e-95eb-d6e78f887fcf
4a5f132b-81c2-49ba-8934-0257f98e5a54
d2e80ec0-9bd3-4801-bada-65a695f7dc92
1d59c9a1-090c-4333-b2af-4154d5edd9c2
c0ff03ac-a195-46a7-a507-5c1cf5c7f668
3d029511-2627-458a-a06b-e2c8e86d26b4
846e76bc-b96f-40aa-a30f-b6d2f84fd7f0
55158cdb-07cc-4c02-8f44-a19ec2aeca06
b9a00edd-9e28-4549-8f13-2634577c1276
9aeccb7b-e629-4f0a-ab81-5300c7c433c0
a2f073a8-f0b0-4f58-a79a-2f22131016b1
f5cd9efa-d735-41d5-a90f-ba83357f2e40
a40d0c35-8232-42db-b5a2-2118d69d6c41
41e42d9c-efe9-45c5-9f8e-a2f5f441dd73
efe99726-557a-4595-90e8-9c0fc3ee0c20
6ce7f634-5c44-4818-a055-cb32dca20738
d1243377-9aed-4abf-8207-90b47840a48f
bd5be404-6a8b-402a-8a6f-856eaa054664
dc95a098-107f-46c9-8d36-23cf575a319f
5166cb9a-20ff-4d5c-b224-88f0b2276398
954559e9-5f8c-4f9b-b2bc-36ff23f18d4a

that looks great so replace the echo with the real command:

nmcli connection delete uuid $i

this will lead to the follwing final command which is identical to the one on the top of the article but just with some line breaks for better readability.

$ for i in `nmcli c | \
grep "never" | \
grep -o -- "[0-9a-fA-F]\{8\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{4\}-[0-9a-fA-F]\{12\}"` ; \
do nmcli connection delete uuid $i ; \
done

Now I only have a few connections in my network manager so that I can easily switch connections from the command line depending on where I am:

$ nmcli c
NAME UUID TYPE TIMESTAMP-REAL
AndroidAP 375e398a-1b2e-4d2e-b4a5-4c05c904b109 802-11-wireless Fr 02 Jan 2015 16:29:17 CET
FRITZ!Box 7330 SL a431f83e-6a73-4cf1-94ee-4409c19dfdb9 802-11-wireless Do 13 Feb 2014 18:45:48 CET
NAS QNAP ef6a08df-1c7c-48ba-b476-f3ff3b1e2669 802-3-ethernet Mi 15 Apr 2015 11:03:51 CEST
MSDSL2 f2fbd0b7-26f6-4939-aece-c2c9d3621e40 802-11-wireless Do 16 Apr 2015 20:28:22 CEST
shared 806c9806-fadc-472f-8460-318babbd38ea 802-3-ethernet Fr 10 Apr 2015 09:39:50 CEST
Hotel Amsee3 f4ff9826-22d2-4f9b-91d9-8c63dc0f9bd4 802-11-wireless So 29 Mär 2015 11:56:21 CEST
Lummerland 7d2d4d89-9eb7-45c0-a963-a8a240e33100 802-11-wireless Mo 10 Feb 2014 22:35:03 CET
FRITZ!Box Fon WLAN 7270 50d62587-5a48-4635-b30f-fbb1ded20455 802-11-wireless Di 30 Dez 2014 12:23:23 CET
Arcor-362007-L 87a2d11c-3a3d-4b5e-a1b0-cc5b38916035 802-11-wireless Di 14 Apr 2015 17:42:33 CEST
Telekom 60aded97-f697-48c6-8070-91e1b786df82 802-11-wireless Mi 15 Apr 2015 00:14:14 CEST
Wired connection 3 bf70a943-9510-4883-ba64-a4bede826120 802-3-ethernet Do 16 Apr 2015 09:26:08 CEST
FRITZ!Box Fon WLAN 7360 7c428f50-078d-442a-9ba7-268853c888f0 802-11-wireless Mo 11 Feb 2013 23:39:08 CET
test 836af61a-8268-4a16-a5ab-036f3e0cf7e9 802-11-wireless So 15 Sep 2013 21:18:35 CEST
Wired connection 1 30ffcf0e-5181-41b0-b7d4-402de875889f 802-3-ethernet Mo 04 Mär 2013 15:10:44 CET
Telekom_ICE 8763adcc-f27b-441f-9184-594128871351 802-11-wireless So 29 Mär 2015 19:56:35 CEST
Wired connection 2 700ebb58-f049-4ec7-bac6-cd5801a975e2 802-3-ethernet Do 16 Apr 2015 18:11:44 CEST
WLAN-8CA902 3046307e-69a3-4c99-89fb-b79f7f9ee24b 802-11-wireless Mi 01 Apr 2015 11:42:27 CEST
uni-koblenz 8a0d1d51-672c-4ded-ad92-18e27b8215df 802-11-wireless Do 16 Apr 2015 18:11:45 CEST

]]>
https://www.rene-pickhardt.de/cleaning-up-my-network-connections-on-ubuntu-linux-using-the-network-manager-nmcli/feed/ 0
How to learn to learn programming for non techies? https://www.rene-pickhardt.de/how-to-learn-to-learn-programming-for-non-techies/ https://www.rene-pickhardt.de/how-to-learn-to-learn-programming-for-non-techies/#respond Mon, 01 Sep 2014 22:33:41 +0000 http://www.rene-pickhardt.de/?p=1898 Out of my personal offline social network I have been frequently approached by non techies that ask me questions like

  • “I want to learn programming, which language should I learn?”
  • “At what school / course can I participate to learn programming?”
  • “Which book can you recommend if I want to learn programming?”

These questions are in my opinion very tough but I will try to give an answer. Even though a lot of what I will say is probably true for people like me who learned programming just by curiosity while being in high school or for people who wish to make a career in IT engineering I try to aim this article for people with a university degree who already have a certain knowledge of how to approach learning and want to have programming skills as an additional skill but not for their main living.
The outline of this article will be the following.

  • Why do you want to learn programming? What I will assume about you.
  • What does it mean to learn programming?
  • What is really necessary in order to learn programming?
  • Discussing various kind of languages for certain purposes.
  • Your concrete road map.

Why do you want to learn programming? What I will assume about you.

I will assume that you do not want to learn programming because you want to change your major towards IT or that you are suddenly totally interested in programming, computers, nerds and geek talk. I will rather assume that you might be a linguist who realizes that statistical natural language processing might be useful. Or you are a biologist or psychologist and need to automate some work with your experimental data from the lab. You could also be a sociologist and want to do some experiments on the web or study how people behave on the web. Maybe you ended up in a company because jobs in your major are rare and you seek for an additional qualifications. You might have similar reasons but I guess you get the gist of the article.

What does it mean to learn programming? 

Being able to program is a somewhat weird skill. Those who are able to code know clearly what they can expect from another hacker. Yet I have the feeling that people who want to learn programming often have wrong expectations what programming means. So I will try to work on your expectation management.
Know to code is a skill that exists on various levels. Being a good programmer and acquiring programming skills that are really useful for you might be a long road and might require some effort (c.f. peter norvig’s “how to learn programming in 10 years?” ). Still I have to state (in a somewhat arrogant way) that the first step is very easy (at least in theory).
The first step would probably be to learn the syntax – which are the commands or instructions of a programming language. This is simple because programming languages usually have a very small syntax. Learning the syntax and playing around with it should seriously not take you more than a view hours in total. If you consider practice and repetitions this time will be distributed over a couple of days. This is what books with titles like “how to learn XXX in YYY days” which are heavily criticized by Norvig will teach you. The same can be learnt from online courses on http://www.codecademy.com/ The main problem here is that this skill – even though at core it is all you need – seems very useless to a beginner.
In particular once you have acquired this skill you will have the feeling to know as much as before. Compare it with having the force like a Jedi knight. You might feel it but you don’t know how to use those skills yet.
In my oppinion the main problem starts now. In order to make something meaningful with your new skill you will do much more that will distract you from using or learning that skill. You will become frustrated. You will find everything complicated and you most likely will often have the feeling that you don’t know why something does not work as explained and if it finally does you might also not know why it does so at this point.
Common pitfalls for distraction are:

  • Interacting with an Integrated development environment.
  • The same operator in a programming language might have a different semantics depending on the context.
  • Interacting with a compiler or interpreter.
  • (Most likely implicitly) interacting with the operating system
  • Using third party API’s and libraries.
  • Interacting with an editor that has its special properties
  • Doing a typo with all the new syntax receiving error messages that are not telling you “hey you forgot a semicolon here…” but rather look like total chaos.

The problem is these things are almost not documented at all and are just assumed to be learnt implicitly or while talking to people or while being in a classroom. You might want to have a look at my German screen cast where I explain how to run “hello world in c“.

The program is the most basic program and and it involves almost no programming at all it just has all the overhead that was described above. Even though the screen cast is 20 minutes I still omit so much information that you might think: “Why the hell should I learn all about this?” The point I am trying to make is that it is all the other stuff that is complex.
Taking this in mind I sometimes wonder how anyone can learn programming at all anyway? The bad news is. Books won’t tell you all the differences most of the time they don’t even help you with setting up your developing environment so they leave you lost to learn a task which – again – at its core is actually pretty simple.
I would say a subset of being able to program is actually being aware of all these things and being able to distinguish what is happening at what point in time and not becoming confused by all this “magic”.
It already needs some experience to know all of this. The good news is. Once you master these things you know about everything you need in order to move on.

What is really necessary?

Even though you just want to program a little bit I think what will help you most is to bring curiosity about computers in general and the willingness to iterate, fail and play around. Don’t get frustrated if you seem to not understand anything anymore. Abstract and structured thinking is a very helpful skill which you probably bring along when you read this article and have been able to follow until here.
Most likely you will have to move out of your comfort zone. I remember learning 10 finger typing. In the beginning I thought chatting with friends is faster with my old system. I had to move out of my comfort zone to use the new skill and I had to use it in order to get used to it. The same will hold true for programming. In the beginning you might think that stuff from your day to day job go faster the way you have always done them. Take the challange. This time using programming might take longer. But over time it will save you a hack a lot of time.
But moving out of your comfort zone will mean that you have to be open to geek culture. Computer scientists are nerds and they celebrate it. I honestly learnt stuff about hacking from xkcd comics (or similar stuff).
Having courage to try out stuff and play around will be essential but still many people seem to confuse “try and error” with meaning full testing of “what happens actually if I …” (sorry that was so important I had to bold print it and I will repeat it later…). Read the fucking manual (or documentation – even though it is often missing and badly written) will be of tremendous help for you. What you have to learn is to ask as many questions as possible. A good practice is to write down the questions and explaining your problems. While doing so you will understand your problem better and most likely solve the problem on your own. (Ask questions even to this article. I am well aware of the fact, that this article leaves open questions. Go and ask me in the comments as an exercise. (NOW (:)) In general before asking questions googeling is a great idea. This of course is another way of asking questions. Honestly it is a skill to use a search engine. I have seen so many students which are not able to properly use a search engine to solve their problems. Fact is if you want to learn programming almost all questions that you have, have already been asked on the web and are indexed by a search engine. Being aware of this fact and learning to find those answers is a large portion of what you need to become a programmer.
Fiannly you will need a lot of paticence. The complexity of programming comes from the sheer amount of technologies that play together. It will be inevitably that you will be confused. Stay focused take a deep breadth start over again and try once more.

Discussing various kind of languages for certain purposes

There are various kinds of languages which serve a certain purpose. If you are coming from the outside world you will most likely know what language you want to learn. Honestly the syntax is almost the same all the time anyway. What is different are the libraries and APIs. When gaining more experience you will realize that you use tools to remember APIs and at some point in time you have a feeling of what functions already exist in some API and you probably can guess the name or at least google for it. What I am trying to say it doesn’t really matter which language you will take for learning how to program.
Still I think there are three languages that might be particular interesting candidates. All of these language have object oriented concepts. Even though I think object oriented programming is great and probably not so difficult for you. You could probably ignore the object oriented concepts.
c/c++: This language is great for people who want to focus on the understanding part of fundamental stuff and computers. It is certainly most difficult to learn and easy to mess up things. The level of frustration will be highest but the reward will be the biggest. I learnt programming with C first and from there on I could very easily move on to other languages.
Python: I think Python has a very beautiful syntax and it minimizes the distraction elements I wrote about so broadly. Especially for people who want to use programming as an additional skill python is very quick for solving smaller tasks with few overhead on the amount of code that you need. Also the standard API that comes with python is easy to use. Python can be used in the Web, in the shell. So it is probably a very nice compromise among all scripting languages.
Java: I would say Java is one of the widest spread languages. It has many strong concepts and especially a lot of tool chain around it. I did not find any other language that has so much code completion and tools to make your life easier. Still having tools thinking for you while you did not understand something is a difficult thing.
There are other languages and excellent reasons why you would want to learn them. The only language I would not recommend is java script. This language is a mess for various reasons. So unless you want to explicitly do user interfaces on the web I see almost no reason why a non IT person should learn javascript.

Your concrete road map

Let’s finally get concrete now. How to approach this project of learning to program:
10 relatively easy tips for the first week:

  1. Choose a language
  2. Install the IDE, compiler, toolchain, …
  3. Do it the hard way. Install linux (e.g ubuntu), use a fancy editor like emacs or vim and go for a fancy tool chain or version control system like git. Not that you need it. But as I said it forces you out of your comfort zone and brings you in a hacking mindset. This is most important. Loosing the fear of shallow water within usage of computers. You need to loose your fear of technology.
  4. Learn the syntax with some random book / course about how to learn programming or your selected language in 7 days. (Again the book could nowadays also be an online course)
  5. Understand that most books are didactically very bad and don’t separate between concepts, APIs, operating systems and language specific parts. 
  6. Use a cheat sheet for learning the syntax. Not in order to cheat all the time. But it helps you to distinguish between what is syntax and what is all the distracting overhead. just google for “c cheat sheet” or “java cheat sheet” or “python cheat sheet”.
  7. Understand the syntax and remember it by heart.
  8. Try not to read the examples and remember them by heart but try to reproduce them without looking at them. If you can’t read the chapter again and iterate.
  9. Most important: While working through the book / course. Play around! If something worked start asking questions. What happens if I change code here. Try to answer the question and then try it out. Even try to break the code in order to see error messages from the compiler / interpreter. don’t get confused by naming try try try and try again. 
  10. Understand the concept of variable scope – this can also be done best while playing around.

Once you are here and as I said you should not need to long. Go to the next level – The transfer / experience stage.

  1. Ask yourself what is a task that you frequently do by hand (on a computer or elsewhere) which IT people might automate. Try to automate it.
  2. Find some open source project. Look for bugtracker, mailinglist, or IRC channel. Go an contribute. Experience, experience experience. talk to the people. almost every IT person is happy if you try to solve their problems (which they just might not find the time to solve themselves). Tell the people you are a beginner. Chances are pretty high that people will help you out.
  3. Learn blackboxing (after I learned programming with 12 and creating a large scale application with 22 I started to really dig into what computers and programming languages are 4 years ago. I understood many things down to the physical process. yet still most stuff I use is a blackbox to me. It is much less of a blackbox compared to 10 years ago but still most is a blackbox!) You cannot understand everything. Sometimes it is just enogh to know “hey if I enter print(‘a’); the letter a will be outputed to the terminal.” You don’t have to understand how system i/o (input and output) really works. it will be sufficient for you to be able to use it.
  4. Understand what libraries and APIs are. Understand that some instructions and commands that first seem to be typical code (or syntax) of your language are just APIs that you should use as a black box. a typical example is the above mentioned hello world program.
  5. To some degree understand what an operating system is, what its purpose are.
  6. Have a concrete roadmap project. Nothing is more boring that a computer scientist telling you “hey I can automatically calculate square numbers…” I once wrote an blog article explaining with the python language how to find the most frequent words in books or the longest sentence and decide how long it is… (I admit this program was probably to complex for a beginner. but you could still try)  

Finnaly the hard parts

  1. As with all learning tasks find a) a sparing partner and b) as early as possible teach the stuff to others.
  2. Get a feeling on what is important to know while programming.
  3. Listen to the fucking nerd. We sometimes might not communicate well (guess what: Communication is a two way thing. Chances are high you are not asking well ether and not communicating your needs in a good way) But most nerds will be very happy to help you out. They might assume you have too much knowledge which you do not have. If they do, stop them or slow them down. They might loose themselves in details. If they do, try to ask them if this is still relevant to the question and going towards the right direction. Do your job on good communication. Remember your talking to someone from a different subculture (:
  4. Have clear questions and clear goals.

I wish you good luck and a lot of fun while acquiring your new skills. If this article was helpful for you or changed your life please tell me in 10 years or so what kind of amazing things you have built since then. Just put the mark in your calendar right now. You don’t use an electronic calender system yet? Ohhh, much to learn young Padawan (:

]]>
https://www.rene-pickhardt.de/how-to-learn-to-learn-programming-for-non-techies/feed/ 0
Version control of your Linux config with git https://www.rene-pickhardt.de/version-control-of-your-linux-config-with-git/ https://www.rene-pickhardt.de/version-control-of-your-linux-config-with-git/#comments Thu, 24 Apr 2014 17:05:37 +0000 http://www.rene-pickhardt.de/?p=1840 I was just reading through the recent notes of Heinrich which I can recommend to read as well as his old notes. When I stumbled upon the note called Monitor /etc/ using git I was confused. Why would one do this?
So I talked to Heinrich and he said:

“Well you want to monitor changes of your system config. You want to be able to revert them and you don’t want to care about this when you do something.”

I really liked this and thinks its so useful and a smart idea that I wanted to share it with you. Just keep in mind that you don’t push the git repository to some public space since the config files might include a lot of passwords. Also look out for his .gitignore in his case the printer does a lot of automatic changes and is thus ignored. You might have similar settings for your configs.
I hope sharing this was useful for you!

]]>
https://www.rene-pickhardt.de/version-control-of-your-linux-config-with-git/feed/ 4
What should I do with my 10 to 15 year old desktop pcs? https://www.rene-pickhardt.de/what-should-i-do-with-my-10-to-15-year-old-desktop-pcs/ https://www.rene-pickhardt.de/what-should-i-do-with-my-10-to-15-year-old-desktop-pcs/#comments Sat, 18 Jan 2014 12:08:35 +0000 http://www.rene-pickhardt.de/?p=1792 Hey everyone I wonder if you could help me out. I am currently at my parents home and there are some old pcs from the time when I was young (even my first very on pc is among them). They might have been bought between 1997 and 2002 and have single core processors starting from 333 Mhz going up to 1800 Mhz. Memory is also varying between 64 MB to 1 GB as is the hard disk. Those computers need way to much energy, the fan is really loud and so on…
In general the electronic parts of these computers are still in a good shape and they have served a good purpose for a long time. I can imagine many use cases yet looking at ebay you would only be able to sell these computers for 1 euro each.
I kind of refuse to give them away for free or even worse trow them away. But apparently these computers are worth nothing. Which sorry to say this again I refuse to accept. Computing power is an amazing thing.
Does anyone have a cool idea what one could do with them? Maybe install some lite weight Linux and use them to control some hardware or investigate some networking projects. I even considered using them as a file / backup server but this also seems not to be a good idea since the energy consumption as mentioned above is too high and network storage devices which you can buy nowadays seem to fulfill the service much better.
I tried to google for the problem but I only find boring articles without any good ideas. So if anyone of you had some idea this would be highly appreciated.

]]>
https://www.rene-pickhardt.de/what-should-i-do-with-my-10-to-15-year-old-desktop-pcs/feed/ 14
Building an Autocomplete Service in GWT screencast Part 4: Integrating the neo4j Data base https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-4-integrating-the-neo4j-data-base/ https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-4-integrating-the-neo4j-data-base/#comments Thu, 20 Jun 2013 12:38:46 +0000 http://www.rene-pickhardt.de/?p=1640 In this screencast of my series I explain at a very basic level how to integrate a data base to pull data for autocomplete queries. Since we have been working with neo4j at this time I used a neo4j data base. It will be only in the next two parts of this series where I introduce an efficient way of handling the data base (using the context listener of the web server) and building fast indices. So in this lesson the resulting auto complete service will be really slow and impractical to use but I am sure for didactic reasons it is ok to invest 7 minutes for a rather bad design.
Anyway if you want to use the same data set as I used in this screencast you can go to http://data.related-work.net and find the data set as well as a description of the data base schema:

]]>
https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-4-integrating-the-neo4j-data-base/feed/ 2
Building an Autocomplete Service in GWT screencast Part 3: Getting the Server code to send a basic response https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-3-getting-the-server-code-to-send-a-basic-response/ https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-3-getting-the-server-code-to-send-a-basic-response/#comments Mon, 17 Jun 2013 12:20:11 +0000 http://www.rene-pickhardt.de/?p=1626 In this screencast of my series on building an autocomplete service you will learn how to implement a Server servlet in GWT such that autocomplete queries receive a response. In this video the response will always be static and very naive. It will be up to the fourth part of this series which will follow already this week to make the server to something meaningful with the query. This part is rather created to see how the server is supposed to be invoked and what kind of tools and classes are needed. So see this as a preparation for the really interesting stuff.

If you have any questions, suggestions and comments feel free to discuss them.

]]>
https://www.rene-pickhardt.de/building-an-autocomplete-service-in-gwt-screencast-part-3-getting-the-server-code-to-send-a-basic-response/feed/ 1
Create a Screencast in Ubuntu with recordmydesktop and do Soundengineering and post production https://www.rene-pickhardt.de/create-a-screencast-in-ubuntu-with-recordmydesktop-and-do-soundengineering-and-post-production/ https://www.rene-pickhardt.de/create-a-screencast-in-ubuntu-with-recordmydesktop-and-do-soundengineering-and-post-production/#comments Fri, 17 May 2013 18:36:55 +0000 http://www.rene-pickhardt.de/?p=1600 I promised to create some screen casts for the autocomplete service with GWT based on neo4j. After I have created all the screencasts I had to go through quite some hassle in order to do so.
So let me share my experience and toolchain in order to produce a somewhat accaptable screen cast. I am still not happy with the quality of the results and if you have any suggestions feel free to tell me.
First of all you download and install recordmydesktop. Once this is done you can just do a screencast by calling the following line.
recordmydesktop --quick-subsampling --full-shots --no-shared --v_bitrate 2000000 --on-the-fly-encoding
bear in mind that you should have at least a dual core processor if you really want to do video encoding on the fly.
After creating this screencast I realized that my audio and video tracks where not in sync. Even worse. I realized that the video track was always a few seconds shorter than the audio track. At the end of the video when the audio wasn’t finnished yet the last video frame would just be displayed. This might be alright on a 20 second screencast but some of my screencasts where up to 14 minutes and that really sucked! because speaking and video would just not be synchrone anymore.
In another screen cast that I did today I had the problems with the sound not being loud enough so Here is the stack of software that works fine with corrupt .ogv files produced by recordmydesktop.
After searching quite a while and trying several programms (pitivi, Avidemux,oggtools(!),…) that all crashed on the .ogv file that came out of recordmydesktop I reallized that with ffmpeg2theora I could actually extract the audio and video files seperately. to do so just enter:
ffmpeg2theora video.ogv --noaudio
ffmpeg2theora video.ogv --novideo

The problem was the Audiofile. I was able to play it with my Audioplayer but again not able to open it with any editing software but oggconvert!
With this little tool I was able to open the corrupt .oga audio file and save it to an .ogg file (with vorbis audio codec!).
Now after I had a non corrupted audiofile I was able to adjusting the speed of the audiofile adjusting it to the length of the recoreded video with audacity. Easy instructions how to do this are available at http://wiki.audacityteam.org/wiki/Change_Speed Audacity also provided the options for me to amplify the volume of the sound.
Finally I had two files of the same length just need to merge them again. The oggTools again failed to help me on this but there is pitivi which is able to do the job.
So you see if one keeps patient one will reach his goal. Some of the screen casts are already published and more will come. So far I hope you enjoyed reading my experiences and solutions
by the way on wikipedia there are quite some tutorials on how to edit ogg videos.
http://commons.wikimedia.org/wiki/Help:Converting_video

]]>
https://www.rene-pickhardt.de/create-a-screencast-in-ubuntu-with-recordmydesktop-and-do-soundengineering-and-post-production/feed/ 2
How to do a presentation in China? Some of my experiences https://www.rene-pickhardt.de/how-to-do-a-presentation-in-china-some-of-my-experiences/ https://www.rene-pickhardt.de/how-to-do-a-presentation-in-china-some-of-my-experiences/#comments Fri, 02 Nov 2012 08:33:22 +0000 http://www.rene-pickhardt.de/?p=1432 So the culture is different from Western culture we all know that! I am certainly not an expert on China but after living in China for almost 2 years knowing some language and working in a chinese company seeing presentations every week and also visiting over 30 western and chinese companies placed in China I think I have some insights about how you should organize your presentation in China.
Since I recently went to Shanghai in order to to research exchange with Jiaotong University I was about to give a presentation to introduce my institute and me. So here you can find my rather uncommon presentation and some remarks, why some slides where designed in the way they are.
http://www.rene-pickhardt.de/wp-content/uploads/2012/11/ApexLabIntroductionOfWeST.pdf

Guanxi – your relations

First of all I think it is really important to understand that in China everything is related to your relations (http://en.wikipedia.org/wiki/Guanxi). A chinese business card will always name a view of your best and strongest contacts. This is more important than your adress for example. If a conference starts people exchange namecards before they sit down and discuss.
This principle of Guanxi is also reflected in the style presentations are made. Here are some basic rules:

  • Show pictures of people you worked together with
  • Show pictures of groups while you organized events
  • Show pictures of the panels that run events
  • Show your partners (for business not only clients but also people you are buying from or working together with in general)

My way of respecting these principles:

  • I first showed a group picture of our institute!
  • I also showed for almost every project where I could get hold of it pictures of the people that are responsible for the project
  • I did not only show the European research projects our university is in but listed all the different partners and showed logos of them

Family

The second thing is that in China the concept of family is very important. I would say as a rule of thumb if you want to make business with someone in china and you havent been introduced to their family things are not going like you might expect this.
For this reason I have included some slides with a worldmap going further down to the place where I was born and where I studied and where my parents still leave!

Localizing

When I choosed a worldmap I did not only take one with Chinese language but I also took one where china was centered. In my contact data I also put chinese social networks. Remember Twitter, Facebook and many other sites are blocked in China. So if you really want to communicate with chinese people why not getting a QQ number or weibo account?

Design of the slides

You saw this on conferences many times. Chinese people just put a hack a lot of stuff on a slide. I strongly believe this is due to the fact that reading and recognizing Chinese characters is much  faster than western characters. So if your presentation is in Chinese Language don’t be afraid to stuff your slides with information. I have seen many talks by Chinese people that where literally reading word by word what was written on the slides. Where in western countries this is considered bad practice in China this is all right. 

Language

Speaking of Language: Of course if you know some chinese it shows respect if you at least try to include some chinese. I split my presentation in 2 parts. One which was in chinese and one that was in english.

Have an interesting take away message

So in my case I included the fact that we have PhD positions open and scholarships. That our institut is really international and the working language is english. Of course I also included some slides about my past and current research like Graphity and Typology

During the presentation:

In China it is not rude at all if ones cellphone rings and one has more important stuff to do. You as presenter should switch of your phone but you should not be disturbed or annoyed if people in the audience receive phone calls and go out of the room doing that business. This is very common in China.
I am sure there are many more rules on how to hold a presentation in China and maybe I even made some mistakes in my presentation but at least I have the feeling that the reaction was quite positiv. So if you have questions, suggestions and feedback feel free to drop a line I am more than happy to discuss cultural topics!

]]>
https://www.rene-pickhardt.de/how-to-do-a-presentation-in-china-some-of-my-experiences/feed/ 3
Graphity source code and wikipedia raw data is online (neo4j based social news stream framework) https://www.rene-pickhardt.de/graphity-source-code/ https://www.rene-pickhardt.de/graphity-source-code/#comments Mon, 09 Jul 2012 15:43:57 +0000 http://www.rene-pickhardt.de/?p=1377 UPDATE: there is now the source code of an entire graphity server application online!
8 months ago I posted the results of my research about fast retrieval of social news feeds and in particular my graph index graphity. The index is able to serve more than 12 thousand personalized social news streams per second in social networks with several million active users. I was able to show that the system is independent of the node degree or network size. Therefor it scales to graphs of arbitrary size.
Today I am pleased to anounce that our joint work was accepted as a full research paper at IEEE SocialCom conference 2012. The conference will take place in early September 2012 in Amsterdam. As promised before I will now open the source code of Graphity to the community. Its documentation could / and might be improved in future also I am sure that one is even able to use a better data structure for our implementation of the priority queue.
Still the attention from the developer community for Graphity was quite high so maybe the source code is of help to anyone. The source code consists of the entire evaluation framework that we used for our evaluation against other baselines which will also help anyone to reproduce our evaluation.
There is some nice things one can learn in setting up multthreading for time measurements and also how to set up a good logging mechanism.
The code can be found at https://github.com/renepickhardt/graphity-evaluation and the main Algorithm should lie in the file:
https://github.com/renepickhardt/graphity-evaluation/blob/master/src/de/metalcon/neo/evaluation/GraphityBuilder.java
other files of high interest should be:

I did not touch it again over the last couple months and it really has a lot of debugging comments inside. My appologies for this bad practice. I hope you can oversee this by having in mind that I am a mathematician and this was one of my first bigger evaluation projects. In my own interest I promise next time I produce code that will be easier to read / understand and reuse.
Still if you have any questions suggestions or comments feel free to contact me.
The raw data is can be downloaded at:

the format of these files is straight foward:
de-nodeIs.txt has first some ID then a tab and then the title of the wikipedia article this is just necessary if you want to display your data with titles rather than names.
the interesting file is the de-events.log in this file there are 4 columns
timestamp TAB FromNodeID TAB [ToNodeID] TAB U/R/A
So every line tells exactly when an article FromNodeID changes. if only 3 collumns are available and an U is written then the article just changed. Maybe links in the article changed in this case there exists another nodeID in the 3 column and an A or a R for add or remove respectively.
I think processing these files is rather straight forward. With this file you can totally simulate the growth of wikipedia over time. The file is sorted by the 2. column. If you want to use it in our evaluation framework you should sort this by the first column. This can be done on a unix shell in less than 10 minutes with the sort command.
Sorry I cannot publish the paper right now on my blog yet since the camera ready version has to be prepared and checked in to IEEE. But follow me on twitter or subscribe to my newsletter so I can let you know as soon as the entire paper as a pdf is available.

]]>
https://www.rene-pickhardt.de/graphity-source-code/feed/ 7
Building an Autocompletion on GWT with RPC, ContextListener and a Suggest Tree: Part 0 https://www.rene-pickhardt.de/building-an-autocompletion-on-gwt-with-rpc-contextlistener-and-a-suggest-tree-part-0/ https://www.rene-pickhardt.de/building-an-autocompletion-on-gwt-with-rpc-contextlistener-and-a-suggest-tree-part-0/#comments Wed, 13 Jun 2012 13:15:29 +0000 http://www.rene-pickhardt.de/?p=1360 Over the last weeks there was quite some quality programming time for me. First of all I built some indices on the typology data base in which way I was able to increase the retrieval speed of typology by a factor of over 1000 which is something that rarely happens in computer science. I will blog about this soon. But heaving those techniques at hand I also used them to built a better auto completion for the search function of my online social network metalcon.de.
The search functionality is not deployed to the real site yet. But on the demo page you can find a demo showing how the completion is helping you typing. Right now the network requests are faster than google search (which I admit it is quite easy if you only have to handle a request a second and also have a much smaller concept space).  Still I was amazed by the ease and beauty of the program and the fact that the suggestions for autocompletion are actually more accurate than our current data base search. So feel free to have a look at the demo:
http://134.93.129.135:8080/wiki.html
Right now it consists of about 150 thousand concepts which come from 4 different data sources (Metal Bands, Metal records, Tracks and Germen venues for Heavy metal) I am pretty sure that increasing the size of the concept space by 2 orders of magnitude should not be a problem. And if everything works out fine I will be able to test this hypothesis on my joint project related work which will have a data base with at least 1 mio. concepts that need to be autocompleted.
Even though everyting I used but the ContextListener and my small but effective caching strategy can be found at http://developer-resource.blogspot.de/2008/07/google-web-toolkit-suggest-box-rpc.html and the data structure (suggest tree) is open source and can be found at http://sourceforge.net/projects/suggesttree/ I am planning to produce a series of screencasts and release the source code of my implementation together with some test data over the next weeks in order to spread the knowledge of how to built strong auto completion engines. The planned structure of these articles will be:

part 1: introduction of which parts exist and where to find them

  • Set up a gwt project
  • Erease all files that are not required
  • Create a basic Design

part 2: AutoComplete via RPC

  • Neccesary client side Stuff
  • Integration of SuggestBox and Suggest Oracle
  • Setting up the Remote procedure call

part 3: A basic AutoComplete Server

  • show how to fill data with it and where to include it in the autocomplete
  • disclaimer! of not a good solution yet
  • Always the same suggestions

part 4: AutoComplete Pulling suggestions from a data base

  • inlcuding a data base
  • locking the data base for every auto complete http request
  • show how this is a poor design
  • demonstrate low response times speed

part 5: Introducing the context Listener

  • introducing a context listener.
  • demonstrate lacks in speed with every network request

part 6: Introducing a fast Index (Suggest Tree)

  • inlcude the suggest tree
  • demonstrate increased speed

part 7: Introducing client side caching and formatting

  • introducing caching
  • demonstrate no network traffic for cached completions

not covered topics: (but for some points happy for hints)

  • on user login: create personalized suggest tree save in some context data structure
  • merging from personalized AND gobal index (google will only display 2 or 3 personalized results)
  • index compression
  • schedualing / caching / precalculation of index
  • not prefix retrieval (merging?)
  • css of retrieval box
  • parallel architectures for searching
]]>
https://www.rene-pickhardt.de/building-an-autocompletion-on-gwt-with-rpc-contextlistener-and-a-suggest-tree-part-0/feed/ 6