planet.opensource.dk
Martin Pihl: Love and Marriage
So, I actually managed to get married. Yeah, can you believe that?
It wasn’t without hazzles though, and the missy was not happy with me all the time either.
You know, I juuuust had to check my e-mail before the ceremony…:
(Click to enlarge)
I’m running my own Ubuntu Solution Provider company in Denmark, but managed an incredible three days off for our honeymoon. I’m so romantic…
(Photography by Rune Mokastet)Related posts
Kristian Nielsen: Micro-benchmarking pthread_cond_broadcast()
In my work on group commit for MariaDB, I have the following situation:
A group of threads are going to participate in group commit. This means that one of the threads, called the group leader, will run an fsync() for all of them, while the other threads wait. Once the group leader is done, it needs to wake up all of the other threads.
The obvious way to do this is to have the group leader call pthread_cond_broadcast() on a condition that the other threads are waiting for with pthread_cond_wait():
bool wakeup= false; pthread_cond_t wakeup_cond; pthread_mutex_t wakeup_mutexWaiter: pthread_mutex_lock(&wakeup_mutex); while (!wakeup) pthread_cond_wait(&wakeup_cond, &wakeup_mutex); pthread_mutex_unlock(&wakeup_mutex); // Continue processing after group commit is now done.
Group leader:
pthread_mutex_lock(&wakeup_mutex); wakeup= true; pthread_cond_broadcast(&wakeup_cond); pthread_mutex_unlock(&wakeup_mutex);Note the association of the condition with a mutex. This association is inherent in the way pthread condition variables work. The mutex must be locked when calling into pthread_mutex_wait(), and will be obtained again before the call returns. (Check the man page for pthread_cond_wait() for details).
Now, when I think about how these condition variables work, something strikes me as somewhat odd.
The idea is that the broadcast signals every waiting thread to wake up. However, because of the associated mutex, only one thread will actually be able to wake up; this thread will obtain a lock on the mutex, and all other to-be-awoken threads will now have to wait for this mutex! Only after the first thread releases this mutex will the next thread wakeup holding the mutex, then after releasing the third thread can wake up, and so on.
So if we have say 100 threads waiting, the last one will have to wait for the first 99 threads to each be scheduled and each release the mutex, one after the other in a completely serialised fashion.
But what I really want is to just let them all run at once in parallel (or at least as many as my machine has spare cores for). There is another way to achieve this, by simply using a separate condition and mutex for each thread, and have the group leader signal each one individually:
Waiter: pthread_mutex_lock(&me->wakeup_mutex); while (!me->wakeup) pthread_cond_wait(&me->wakeup_cond, &me->wakeup_mutex); pthread_mutex_unlock(&me->wakeup_mutex);
Group leader:
for waiter in <all waiters> pthread_mutex_lock(&waiter->wakeup_mutex); waiter->wakeup= true; pthread_cond_signal(&wakeup_cond); pthread_mutex_unlock(&wakeup_mutex);This way, every waiter is free to start running as soon as woken up by the leader; no waiters have to wait for one another. This seems advantageous, especially as number of cores increases (rumours are that 48 core machines are becoming commodity).
"Seems" advantageous. But is it really? Let us micro-benchmark it.
For this, I start up 5000 threads. Each thread goes to wait on a condition, either a single shared one, or distinct in each thread. The main program then signals the threads to wakeup, either with a single pthread_cond_broadcast(), or with one pthread_cond_signal() per thread. Each thread records the time they woke up, and the main program collects these times and computes how long it took between starting to signal the condition(s) and wakeup of the last thread. (Here is the full C source code for the test program).
I ran the program on an Intel quad Core i7 with hyperthreading enabled, the
most parallel machine I have easy access to. The results is the following:
pthread_cond_broadcast():
46.9 msec
pthread_cond_signal():
17.6 msec
Conclusion: pthread_cond_broadcast() is slower, as I
speculated. I would expect the effect to be more pronounced on systems with
more cores; it would be interesting if readers with access to such systems
could try the test program and comment below on the results.
Alexander Færøy: The Camp 2010: Slides
Just a short update: I have just uploaded the slides from the two talks I did at The Camp this year.
The first talk was a lightning talk about what is going on behind the scenes of a large IRC network as seen from an operator’s perspective. You probably won’t get much knowledge from the slides unless you were there.
The slides were made in Apple’s Keynote application and it’s absolutely horrible. I’m back on Beamer and LaTeX again.
The second talk was an introduction to Nokia’s Qt framework and how to write cross-platform GUI applications for both mobile devices and desktops (focus was on Linux and OS X only though). The presentation was followed up by a small workshop with roughly ten participants. We wrote a simple, yet functional, weather application that pulled weather data using Yahoo’s weather API and displayed it with fancy icons that changed dynamically depending on the current weather condition. It went really well and I believe that everyone who participated will now know how to get started with writing GUI applications using Qt.
Both slides are available from https://secure.0x90.dk/~ahf/talks/.
It was also great for me personally to try to do a workshop with a mixture of people with both coding experience and some with no coding experience at all. I hope that I will find opportunities to do this again in the near future. It was much more interesting than the regular presentations that I have done over the past years.
I’ve also had to find some personal projects now that I’m a full time coding monkey. Right now they includes a Qt “social” library that implements stuff like OAuth, the Twitter API, the Facebook API and so forth. I’m going to put it online under a two-clause BSD license when I feel like it’s ready for public review.
And yes, I know. I should stop spending time on this social bullshit and start working on some useful stuff, but hey, it’s interesting and I’ll probably get back to some of the much more serious IRC related projects soon’ish.
I have also been looking into what I can do on the iPad device, but currently it’s purely something I use on the train on my way to and from work every day. I’d guess that 80% of the uptime of the device is spend on Angry Birds and Amazon’s Kindle application. Both are nice applications and I hope that I’ll never have to order books made of paper again. I used to hate ebooks, but reading on the iPad seems to work and it doesn’t annoy me too much. Also, it’s lovely to have your entire bookshelf with you in your bag. I hope to try one of Amazon’s dedicated Kindle devices at some point. I can imagine it’s better at displaying text than the iPad.
Final note: I’ve joined BSD-dk’s board of trustees. It’ll be interesting to see what we can do there over the next year.
I hope that everyone has had an interesting summer.
Martin Pihl: OpenOffice i stor fremgang
I april 2009 deltog jeg i en OpenOffice konference i Odense arrangeret af OSL. Og torsdag den 9. september er der sørme en OpenOffice igen i Valby, som er gratis og åben for alle (læs mere hos OSL her).
Selvom vi endnu må vente og se, hvordan konferencen forløber, fornemmer jeg allerede nu, at interessen er eksploderet. Allerede ved sidste konference kunne jeg mærke en stor forskel fra foregående år, men 2010 har vist sig at være et afgørende år, og jeg er spændt på, hvordan det giver sig udslag på konferencen.
I år deltager jeg dog ikke som community-medlem i Ubuntu Danmark standen. Sidste år startede jeg som bekendt en virksomhed, O-Biz, der fokuserer primært på Linux (Ubuntu) ydelser som support, tilpasning og rådgivning til både servermiljø og desktop, og O-Biz har nemlig sin egen stand.
Men O-Biz beskæftiger sig også med andre Open Source produkter, og her er OpenOffice en naturlig del af forretningen.
Hvis man er interesseret i, hvad vi kan tilbyde af OpenOffice ydelser som undervisning eller integration, så kom endelig forbi standen. Og naturligvis også hvis du overvejer Ubuntu i din virksomhed, hvad enten det er server eller desktop, vi snakker om
Related posts
Peter Toft: Hvem hacker min maskine?
Poul-Henning Kamp: SW-pat: panik før lukketid ?
Peter Makholm: Overformynderiet
Efter min mening Københavns mest imponerende indgangsparti:
“Desvære” blev Overformynderiet nedlagt i 1982 og nu huser bygningen Arbejdsmarkedsstyrelsen. Heldigvis har de ikke sandblæst det gamle indgangsparti. Selvfølgelig er der parkering forbudt udenfor… Tænk at bilister selv skulle kunne gætte sig til det:
Poul-Henning Kamp: email fra Sct. Peter
Peter Toft: Endnu en super god hjemmeside: Hvemkommer.dk
Poul-Henning Kamp: Drupalcon på turistklasse
Søren Bredlund Caspersen: Ubuntu Global Jam i København
Så er det igen tid til Ubuntu Global Jam.
Endnu engang mødes Ubuntu og Fri Software entusiaster for at arbejde med at forbedre Ubuntu. I weekenden d. 27. – 29. august går det løse over hele kloden – og vi er også med i Danmark.
Vi mødes i København i Labitat, men det er også muligt at være med hjemmefra med en internet forbindelse.
Man bestemmer selv hvad man har lyst til at arbejde med, men Bugs og oversættelse er de mest oplagte kandidater – det er lige til at gå til.
Så hvis du har lyst til at være med, så kom forbi H. C. Ørstedsvej i weekenden, eller log på vores irc kanal og lad os snakke om hvad du har lyst til at arbejde med.
Der er også en Facebook begivenhed man kan tilmelde sig, så vi har et gæt på hvor mange der deltager.
Og så har Mark i øvrigt annonceret at Ubuntu 11.04 får kælenavnet Natty Narwhal (hvilket kan oversættes til Fikse Narhval). Det var i hvert fald gået hen over mit hoved – men jeg har også haft travlt med mange ting de sidste par dage…
Related posts:
Peter Toft: Virker din netbank?
Peter Larsen: Backup til clouden.. Her er et eksempel på hvordan vi gør, hvad gør i?
Jeg har længe brugt Bacula til backup af maskiner. Det kører faktisk smertefrit, selvom opsætningen er lidt langhåret til tider, og man måske savner et gui til opsætningen.
Anyways, min kærlighed er netop faldet på Bacula, fordi det kan linux, freebsd og windows (og mac, men der bruger vi andre ting).
Jeg har med tiden lavet et lille script til at dumpe configfiler ud med kodeord, og opsætning, så det er relativt simpelt at tilføje nye hosts. man kunne sikkert udvide det lidt med en mysql man bare kunne tilføje hosts i, og så dumpe configs ud fra. Det må blive mit næste projekt :)
Bacula laver full, incredimental og differential backups.. Det er open source, det er under udvikling.. Og virker.
Nu har jeg længe kørt det med lokalt storage, og jo mere skrammel vi har, jo mere plads skal vi bruge, for at holde sikkerhedskopier for samme periode som altid. Maskineparken bliver større og jo mere strøm bruger vi. Der er kun en vinder, og det er ikke mig.
Nu tester jeg så et setup med Amazon S3 som lagerplads, dels fordi det er billigt (strøm er dyrt), men også fordi det opfylder en af mine vigtige krav om offsite backup. Vores setup består af en lokal virtuel maskine som bacula server, s3fs som fuse filsystem mount, og lidt hacks og wacks for at få det til at køre.
Og det kører faktisk fint, min største hage ved det er, at disken først skrives lokalt, og så synces ud (hvor reelt man bare står og venter på ok fra filskrivningen) dette kunne sikkert laves mere fikst. S3fs cacher de åbnet filer lokalt, og man kan vælge at ryde op i dem selv, evt med et par dages cache så man dermed minimerer behovet for at have stor lokal disk tilknyttet.
Så, hvordan kommer du igang?
Start med en bacula der virker, og sæt den op til at tage backup til en lokal disk. Når det virker, så er du klar til at lege med Amazon S3.
Opret en konto hos Amazon S3. Lav en “bucket” på US standard lokationen (r177 virker kun med denne lokaltion)
Installation af s3fs
jeg tager udgangspunkt i FreeBSD, og i r177 af s3fs, da det er denne som pt er i ports.
Start med at find s3fs, det er i ports i dette dir /usr/ports/sysutils/fusefs-s3fs
Du laver nu en “make patch” som udfører patchprocessen og nu er pakken klar til compiling, men vi skal lige tilføje lidt extra detaljer som består af en diff som du kan finde her:
[code]
--- s3fs.cpp.orig 2010-02-05 03:45:59.000000000 +0300 +++ s3fs.cpp 2010-05-14 20:51:44.000000000 +0400 @@ -345,7 +345,7 @@ static string AWSAccessKeyId; static string AWSSecretAccessKey; static string host = "http://s3.amazonaws.com";; -static mode_t root_mode = 0; +static struct stat root_stat; static string service_path = "/"; // if .size()==0 then local file cache is disabled @@ -807,8 +807,7 @@ cout << "getattr[path=" << path << "]" << endl; memset(stbuf, 0, sizeof(struct stat)); if (strcmp(path, "/") == 0) { - stbuf->st_nlink = 1; // see fuse faq - stbuf->st_mode = root_mode | S_IFDIR; + *stbuf = root_stat; return 0; } @@ -1278,6 +1277,10 @@ s3fs_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi) { //cout << "readdir:"<< " path="<< path << endl; + // add .. and . + filler(buf, ".", NULL, 0); + filler(buf, "..", NULL, 0); + string NextMarker; string IsTruncated("true"); @@ -1593,7 +1596,7 @@ struct stat buf; // its the mountpoint... what is its mode? if (stat(arg, &buf) != -1) { - root_mode = buf.st_mode; + root_stat = buf; } } }[/code]
Dette er for at få . og .. med på directory listingen, et nødvendigt onde, da bacula bruger det til at definerer at directory eksisterer.
Du laver nu en normal “make install” og laver lidt kaffe..
.. husk at load kldmodulet, “kldload /usr/local/modules/fuse.ko”, du er nu klar til at loade din “bucket”.
Lav et dir som du ønsker at bruge til din bucket, feks /usr/local/backup, og find dit accessKeyId og secretAccessKey under dit amazon login.
Du er nu klar.
Komandoen for at attache drevet er:
/usr/local/sbin/s3fs mybucket -o readwrite_timeout=86400 -o use_cache=/usr/local/s3fs-tmp -oallow_other -o accessKeyId=[amazonid] -o secretAccessKey=[amazonsecret] /usr/local/backup
eller det burde det være, i virkeligheden har jeg haft bedst resultat ved at slå debugging til, og bare lade det være på, det ser sådan her ud:
/usr/local/sbin/s3fs mybucket -d -o readwrite_timeout=86400 -o use_cache=/usr/local/s3fs-tmp -oallow_other -o accessKeyId=[amazonid] -o secretAccessKey=[amazonsecret] /usr/local/backup &
Sidste fintuning..
Du skal ændre din pool (eller pools) i bacula til at splitte dine filer op, dette kan også have betydning for labelformat, så hermed min definition af labelformat og splitting. Dette skal ske pga begrænsning hos Amazon S3, jeg har så valgt en begrænsning der er lidt lavere for at speede evt fragmentering af filer op.. Er dog i tvivl om det har en reel effekt.. Grænsen på Amazon S3 er svjv 5 GB.
Maximum Volume Jobs = 1 # One diskfile / backup
LabelFormat = “${Client}_${Job:s/ /_/g}_${Year}-${Month:p/2/0/r}-${Day:p/2/0/r}-${Hour:p/2/0/r}-${Minute:p/2/0/r}-${Second:p/2/0/r}_${Level}”
Maximum Volume Bytes = 500000000;
Husk at genstarte din bacula..
Sådan, du er nu klar til at slå det hele til..
Tip: skulle din s3fs lukke af en årsag, så er det nok smart at den automatisk starter igen. Du kan derfor med fordel lave en lille bash ting som tjekker for en given fil på s3fs drevet hver time, og genopretter forbindelsen hvis den ikke er der mere..
Tip2: Det kan være en fordel at lave et subdir til hver maskine, da s3fs tilsyneladende laver en eksklusiv lock på diret under upload.
Martin Pihl: Unity in action
Jeg har tidligere skrevet om det nye interface (shell) i Ubuntu 10.10 Netbook Edition kaldet Unity, og her kunne man også se screenshots af det.
Men der er også lavet en video, som demonstrerer Unity, hvor du kan få et nærmere kig på Unity in action:
Related posts
Poul-Henning Kamp: Bluffer Google ?
Martin Pihl: Video-chat i Gmail nu også til Ubuntu og Debian
Langt om længe. Det må kommentaren være. Men nu er det da omsider muligt, for Debian baserede Linux-distroer som Ubuntu, at benytte video-chatten i Gmail.
Der er gået så lang tid, så de fleste har jo nok fundet andre måder at benytte videochat på, men ikke desto er det da glæde, at det trods alt er muligt nu.
For mig (og min virksomhed) der bruger Google Apps i det daglige, betyder det at vi kan skære et kommunikationsværktøj væk, og derved kan nøjes med at bruge et af de værktøjer vi allerede på forhånd bruger.
Understøttelse for RPM-baserede distros kommer snart.
Related posts
Martin Pihl: Ubuntu One på Windows
Tidligere på året så det ud til, at der ville komme en Ubuntu One klient til Windows i løbet af sommeren – senest. Men den kom aldrig, og de mange brugere som har brug for at synkronisere filer via Ubuntu One mellem Windows og Ubuntu, har måtte benytte alternative metoder.
Men nu skulle det være vist: Med Ubuntu 10.10 som udkommer den 10. okober, lancerer Ubuntu One også en masse forbedringer – herunder en Windows klient.
Det skriver de selv på deres blog, hvor også en feature som musik streaming til Android og iPhone telefoner bliver en ny mulighed. Køber man sin musik via Ubuntu One Music Store i Rythmbox, bliver dit musik ikke alene synkroniseret mellem alle dine PC’ere, men det bliver altså også muligt at streame det direkte til smartphonen.
Alt i alt ser det ud til, at der kommer mange og tiltrængte forbedringer til Ubuntu One til oktober, som bl.a. bedre interface med lettere sign-up, mere information om sit data-abonnement og meget andet. Du kan læse et Roadmap på deres wiki her.
Related posts
Henrik Lund Kramshøj: Network flow analysis by Michael W. Lucas, review
Network Flow Analysis by Michael W. Lucas
ISBN: 978-1-59327-203-6
Paperback: 224 pages
Publisher: No Starch Press June 2010
August 2010
Fantastic and very complete information about network flows
ContentThis book is a easy guide to the world of netflow logging and analysis. The content ranges from basic configuration of flow logging and easy customer friendly graphing methods to detailed custom reporting features in the software presented.
While this book does not cover each and every netflow tool available it has a complete walk through allowing you to get started and immediately produce important information for decision makers and troubleshooting.
This book also cover some details that a lot of beginning network people haven't noticed yet, but which are critical for doing netflow analysis. Things like ICMP types and codes and defining what a flow is. Michael also presents filtering and does so while showing you how to build these from simple primitives into fully working and usable examples that you can reuse in production.
The chapters about reporting both show textual representations, hard numbers, and nice graphing tools - suitable for management and others not needing the same level of detail. While showing reporting he not only show the reference, which options are available, but does interpretation of the sample reports.
The book finishes strong by listing common use cases for netflow analysis and if you reach this level in your own network you will have improved things a lot.
Target audienceFocus in this book is on making use of data available from network devices and thus the network administrator is the one doing the actual work. If you are a decision maker you should buy this book for your network guy and benefit from the awesome output he will generate.
You will need a bit of effort if you are not skilled in running tools from the command line, and setting up the tools can seem hard. Fortunately Michael Lucas has already selected a fine list of tools and how to install those.
The strategy of the book is to get you up and running with netflow easily which really works. Then later when you have seen the benefit from netflow you can dig deeper and deeper into reporting and advanced filtering of the data collected.
To summarize the Good stuff- Short - this book is easy to read and short
- Practical - if you follow the strategy and layout you will get going quickly
- Very advanced and complete - given the length of the book it really has a lot of links and references
The subject of netflow is hard to ease into and there are some great tools not described. If possible I would enjoy a follow up book that would connect netflow, intrusion detection, syslogging and monitoring with the same detail - using some selected tools.
ConclusionThis book is mandatory reading for network people, even if they already use netflow. There are sure to be tips and hints that you will enjoy. I read this book in a few days, but I will use the knowledge gained for years to come.
Links- http://nostarch.com/networkflow.htm book web page and links to more resources
David Zeuthen: gdbus(1) bash completion
Enjoy!
