c0t0d0s0.org

As i got the question on the Solaris 11 Techdays: As reported by the Oracle Tech Network Solaris 11 is now in evaluation under the Operating System Protection Profile using the extended packages Advanced Management, Extended Identification and Authentication, Label Security, and Virtualization at level EAL4+.

Ich möchte noch mal einen Veranstaltungshinweise loswerden. Mein Kollege Michael Faerber organisiert für den 8.3 ein Oracle Breakfast in Düsseldorf. Das ist quasi "Vorträge mit Mampf". Zwei Vorträge sind für dieses Event geplannt. Von 09:15 bis 10:30 "Einsatz und Administration von LDOMs" und von 10:45 "Datenmanagement mit ZFS unter Solaris 11". Bei beiden Vorträgen wird es nur wenig Folien, aber dafür sehr viele Livevorführungen am Objekt geben. Beide Vorträge werde ich halten. Wenn Ihr kommen möchtet, schickt bitte eine Mail an oraclebreakfast_dus@c0t0d0s0.org. Da ist ein Forwarder an den Kollegen hinter. Anmeldeschluss ist der 6. März. Weiss nicht ob es der Kollege so toll findet, wenn ich seine Mailadresse hier poste

Für die Solaris 11 Tech Days am 28. Februar in Zürich kann man sich übrigens immer noch anmelden: Solaris 11 Tech Days 2012

It's a few days since the event in Munich, the last one in the series in Germany. The last two weeks were really cool. There were many many people at the Solaris 11 Techdays 2012. That are the moments where i really love my job. Speaking in front of an interested audience about technology. I hope that Zurich will be as great as the events in Germany, but after the last two weeks i have no doubts about it.

That said we had really luck with the dates: A day after the event in Munich snow got a major problem there and as you may know flying to or from FRA is a major pain in the asymmetric photons at the moment due to the apron.

In the ZFS Storage Appliance we have little nice feature enabling you to do migrations of data in the background. It's called Shadow Migration. It's a really useful feature. Imagine you have a RAIDZ. After a time you recognize that RAIDZ wasn't a good decision for your workload and RAID10 would be much better choice. But how to transform it into a RAID10 and how to do it with minimal interruption? You can do this with the Shadow Migration feature. With the Shadow Migration feature, you can migrate the data from one local or remote filesystem to another, while you are already accessing the new one to get the data on the old ZFS filesystem. This feature is available in Solaris 11 as well.

For this demonstration we will use two zfs pools consisting out of files. So we have to create the files first:

root@test:/test/brainslug# mkfile 128m source1
root@test:/test/brainslug# mkfile 128m source2
root@test:/test/brainslug# mkfile 128m source3
root@test:/test/brainslug# mkfile 128m source4
root@test:/test/brainslug# mkfile 128m target1 
root@test:/test/brainslug# mkfile 128m target2
root@test:/test/brainslug# mkfile 128m target3
root@test:/test/brainslug# mkfile 128m target4
root@test:/test/brainslug# mkfile 128m target5
root@test:/test/brainslug# mkfile 128m target6

Now the pools are created. At first our RAIDZ pool consisting out of 4 files. It's named source

root@test:/# zpool create source raidz \
 /test/brainslug/source1 \
 /test/brainslug/source2 \
 /test/brainslug/source3 \
 /test/brainslug/source4

The second one is the future target of the shadow migration. It consists out of six "disks"

root@test:/# zpool create target
  mirror /test/brainslug/target1 /test/brainslug/target2 \
  mirror /test/brainslug/target3  /test/brainslug/target4 \
  mirror /test/brainslug/target5 /test/brainslug/target6

When you did a basic install, the tools and daemons needed for shadow-migration are not included. You have to install them and enable the shadowd afterwards:

root@test:/test/brainslug# pkg install shadow-migration
root@test:/test/brainslug# svcadm enable shadowd

Now you should see the shadowd daemon running.

root@test:/test/brainslug# ps -ef | grep "shadow"
    root  3292     1   0 14:32:33 ?           0:03 /usr/lib/fs/shadowd

Okay … to test the shadow migration we create a filesystem in the source pool:

root@test:/test/brainslug# zfs create source/somestuff

Now we have to fill this file with a some data. Let's create some play data.

root@test:/test/brainslug# dd if=/dev/urandom of=myfile bs=1024 count=300000                        
300000+0 records in
300000+0 records out
root@test:/test/brainslug# mkdir demodata
root@test:/test/brainslug# cd demodata
root@test:/test/brainslug/demodata# split -b 128k -a 5  ../myfile

This should yield a significant number of 128k files. Now we copy them to the newly created filesystem source/somestuffWe will copy the files into the zfs filesystem posing as our old filesystem:

root@test:/test/brainslug/demodata# cp * /source/somestuff/
root@test:/test/brainslug/demodata# cd /
root@test:/# zfs list source
NAME    USED  AVAIL  REFER  MOUNTPOINT
source  294M  42,1M  46,4K  /source

Just to have something to compare, you could simply count the files and calculate the md5 checksum of a file.

root@test:/# ls -l /source/somestuff | wc -l
    2345
root@test:/# md5sum /source/somestuff/xaadmd
3fb4a6be2f93c3d93998db52061244aa  /source/somestuff/xaadmd

Shadow migration will only works, when the source filesystem read-only. So we have to put the source filesystem into such a state:

root@test:/# zfs set readonly=on source/somestuff

Okay, now let's configure the shadow migration:

root@test:/# zfs create -o shadow=file:///source/somestuff \
target/newlocationforsomestuff

That's all. The command may take some moments to get back. The migration of data starts right in the moment you create the new filesystem. It runs in the background and starts to copy all data to the new filesystem. Important to know: You can do shadow migration via NFS as well and it can be an UFS filesystem as well. you just have to declare the source of the shadow migration like nfs://fileserver/directory

Okay. With shadowstat we can check the process of migration.

root@test:/# shadowstat    
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  25,5M   -       -       00:01:10

The cool think about shadow migration is: You can already use the new filesystem. Despite the fact that the migration is still running, you will already see all files and when you access one file it will be migrated in the moment you access the file on the new filesystem. You don't have to wait with the access, until the block would be migrated by the normal background migration. When you try to access data, that isn't already migrated, it's migrated in the moment you access it in the new filesystem.

root@test:/# md5sum /target/newlocationforsomestuff/xaadmd               
3fb4a6be2f93c3d93998db52061244aa  /target/newlocationforsomestuff/xaadmd
root@test:/# ls -l /target/newlocationforsomestuff | wc -l                       
    2345

Afterwards it proceeds with the further migration of all data in the pool. You can observe that with the shadowstat command.

root@test:/# shadowstat 
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  97,8M   -       -       00:01:50
target/newlocationforsomestuff  128M    -       -       00:02:00
target/newlocationforsomestuff  147M    -       -       00:02:10
target/newlocationforsomestuff  165M    -       -       00:02:20
target/newlocationforsomestuff  186M    -       -       00:02:30
target/newlocationforsomestuff  202M    -       -       00:02:40
target/newlocationforsomestuff  211M    -       -       00:02:50
target/newlocationforsomestuff  224M    -       -       00:03:00
target/newlocationforsomestuff  236M    -       -       00:03:10
target/newlocationforsomestuff  243M    -       -       00:03:20
target/newlocationforsomestuff  249M    -       -       00:03:30
target/newlocationforsomestuff  256M    -       -       00:03:40
target/newlocationforsomestuff  260M    -       -       00:03:50
target/newlocationforsomestuff  266M    -       -       00:04:00
target/newlocationforsomestuff  272M    -       -       00:04:10
target/newlocationforsomestuff  278M    -       -       00:04:20
target/newlocationforsomestuff  286M    -       -       00:04:30
No migrations in progress

root@test:/#

Successfuly migrated.

Do you want to learn more?

Docs
docs.oracle.com: Migrating ZFS File Systems"
docs.oracle.com: Migrating File System Data to ZFS File Systems

Blogs
blogs.oracle.com: What is Shadow Migration
blogs.oracle.com: Shadow Migration Internals

... bleiben noch sechs Veranstaltungen (man kann sich noch anmelden). Der Event in Düsseldorf war schon mal ein guter Auftakt. Viele Leute da, viele Fragen gehabt in den Kaffeepausen. Morgen nun Stuttgart ...

Fand meinen Vortrag ein wenig holprig, zuviel Reuse, bin irgendwie nicht in den Flow gekommen. Habe meinen Vortrag aber für die nächsten Veranstaltungen noch ein wenig umgebaut. Und bevor jemand fragt: Ja ... die Slideanzahl ist noch zweistellig Hab doch nur 30 Minuten ....

PS: Mag mal jemand den Flughafenbetreibern in DUS sagen, das ein Hinweisschild ganz nett wäre, das es da einen Starbucks oben bei der Besucherterasse gibt? Ich bin stumpf dran vorbeigelaufen.

Ich möchte nochmal auf die nächste Woche startende Veranstaltungsreihe zum Thema Solaris 11 hinweisen. Es gibt zwar schon viele Anmeldungen aber ich will "die Hütte voll sehen" . Mehr Informationen sowie eine Agenda dazu findet ihr hier.

Im Februar läuft eine Reihe von Events in Deutschland und der Schweiz zum Thema Solaris 11. Die Events versprechen technisch sehr interessant werden, da die Sprecher jeweils sehr tief in der Materie sind. Über Detlef Drewanz - der bei allen Events dabei ist - muss ich seit dem Containerleitfaden genauso wie über Uli Gräf (der an einigen, aber nicht allen Orten spricht) wohl nichts mehr sagen. Christian Christian Ritzka und Elke Freymann sind ausgewiesene Experten zum Thema OpsCenter. Und ja .. ich halte auch einen Vortrag über Datamanagement in Solaris 11. Und da ich schon zweimal die Frage gesehen habe: Die Veranstaltung ist kostenfrei

08:30 - 09:00	Registrierung
09:00 - 09:15	Begrüßung
09:15 - 10:00	Was ist neu in Oracle Solaris 11 Viele Features, die im Rahmen der Solaris 10 Entwicklung in Solaris 11 Express eingeflossen sind, finden sich auch in Solaris 11 wieder. Diese Präsentation gibt einen Überblick über die neusten Features.
10:00 - 11:00	Oracle Solaris 11 Installation Die wohl herausragendste Eigenschaft von Oracle Solaris 11 ist das neue Package System IPS und der Autoinstaller, die die Installation und das Management von Oracle Solaris 11 vereinfachen. Lernen Sie die neuen Techniken kennen und lassen Sie sich zeigen, wie einfach das Patchen unter Oracle Solaris 11 ist.
11:00 - 11:30	Pause
11:30 - 12:30	Oracle Virtualisierung In Oracle Solaris 11 sind umfangreiche Virtualisierungstechniken integriert. Lernen Sie alles über die neue Netzwerk Virtualisierung in Oracle Solaris 11 und wie sie komplette multi-tier HW Infrastrukturen in einer einzelnen Maschine zusammen mit dem Oracle Virtual Machine framework und Solaris Zonen realisiert werden kann.
12:30 - 13:30	Mittagessen
13:30 - 14:15	Management von IT Infrastrukturen Virtualisierung heist nicht nur "Hypervisor". In diesem Vortrag zeigen wir, wie sich virtualisierte Oracle Solaris 11 Umgebungen zentral verwalten lassen.
14:15 - 14:45	Das Solaris Schulungsprogramm Oracle University stellt zusammen mit unseren Schulungspartnern ein umfassendes Programm zur Vertiefung von Solaris Wissen zur Verfügung. In diesem Vortrag werden die Ausbildungpfade, Kurse und Zertifizierungen für Solaris 11 beleuchtet und verfügbare Lernformen vorgestellt.
14:45 - 15:15	Pause
15:15 - 15:45	Oracle Solaris 11 Datamanagement Oracle Solaris 11 hat umfassende Datamanagement Funktionen integriert. Lernen Sie die neusten ZFS features wie Data Encryption und Deduplikation kennen und wie Sie diese Funktionen über die CIFS-Integration im Kernel auch anderen Plattformen zur Verfügung stellen können.
15:45 - 16:15	Panel, Q&A
16:15 - 16:45	Erfrischungen, Zeit zur Diskussion mit den Experten

Die genaue Agenda mit den Sprechern in den einzelnen Orten und eine Möglichkeit zur Anmeldung findet ihr auf den Eventseiten:

• Düsseldorf 6.2.2012
• Stuttgart 8.2.2012
• Hamburg 9.2.2012
• Potsdam 10.2.2012
• Frankfurt 13.2.2012
• München 14.2.2012
• Zürich 28.2.2012

Um zahlreiches Erscheinen wird gebeten!

Brendan Gregg wrote a really interesting article about tracing ZFS: Activity of the ZFS ARC. Really worth a read.

Buffer Extermination? WTF? Normally i'm seeing wait events like "buffer busy", "log sync" or "db file sequential read" when doing my research in Oracle installation in Top5 events when i'm called because of a situation where the performance is not quite at the level the customer wants. I was sitting in front of the console of an system still using 10g as it's database.

I want to add, that the performance problem had its root somewhere else and was quickly found, however this log wait sparked my interest. A much simpler reason. It was the curiosity afterwards, why there were peaks in the wait event statistics in regular intervals with this wait event i never saw before.

"Buffer exterminate"? WTF ... again. Sounds dangerous. Never saw that before in that list, and than my brain rotated … what the heck is "Buffer Exterminate", i have an idea, something is ringing in my head, but somehow my long-term memory management unit of my brain was unable to stage this information in to current working set. Okay … ask Dr. Google.

Metalink [ID 259137.1] is of great help here. The "buffer exterminate" wait occurs (and can only occur), when the buffer cache is shrunk dynamically and a session wants to access data that is in the granule of the buffer that is chosen by Oracle for removal from the buffer cache. The session wanting the block has to wait until the buffer to be removed has been freed to read it from disk then. You can't simply read the block from disk without waiting, as the block in the granule may represent a new state of the block than the one on the disk an simply reading the one from the disk would yield just outdated data. So you can just wait until the granule has been released.

Before you ask, what a granule is: Oracle doesn't allocate memory in the SGA bytewise, but in so called granules. A granule is 4 MB of memory, when your SGA is up to 1 GB when the instance starts. It's 16 MB when your SGA is larger than 1GB at startup.

In Oracle DB 10g, there is a feature called "Automatic Shared Memory Management". The idea is, that Oracle itself monitors the load and configures the layout of the SGA. I think of automatic means as a very good feature. It's like with manual and automatic gearboxes. Surely, a good driver can accelerate faster with a manual gearbox than with an automatic gearbox, however an automatic gearbox is faster and better than 99% of all drivers. That said, given the existence of behaviour patterns explained by the Dunning-Kruger-effect (h/t to Chris Colomb for hinting me to this interesting phenomenon), 99% of drivers think are part of the 1%. This is especially epidemic in Germany. But back to the issue. It's the same with tuning of systems

You activate the ASMM by setting the parameter SGA_TARGET to a value unequal to zero. Now the system sizes the buffer cache (DB_CACHE_SIZE), shared pool (SHARED_POOL_SIZE), large pool (LARGE_POOL_SIZE) and Java pool (JAVA_POOL_SIZE) automatical within the limit set by SGA_TARGET. If one of the other parameters controling one of the mentioned memory areas is set to a value other than 0, the value is assume as the minimum amount of memory.

Of course: When you have fixed SGA_TARGET and you want to grow one part, another has to shrink. It's obvious that you can't do shrinking simply by throwing the block out of the memory. There may be dirty blocks in that granule(changed blocks that weren't written to disk so far by the database writer to the database file, just to the redo logs).

This works really good and this relieves the admin from investing time to find good values for some of the most important SGA parameters.

However if your database tries to move memory back and forth from one kind of shared memory to another tens of times per hour this is surely not without impact on your database performance. I had such a situation in this case. The system started to move around memory in minute intervals just to move it back a minute or two later. As most automatic systems they will work perfectly within their specification, but you may hit a situation where tries to get most out of a situation with restricted resources, where the SGA is confronted with the situation that all components want more memory and as soon you remove memory from one parts, the other part cries and wants its memory back. That's similar to the argument with your significant other about what's the half of the blanket. Better have two blankets Or to get back to the topic: Have enough SGA ...

How do you find out, how many resizing operations took place? You can look that up by a select statement as described in this blog:

select START_TIME, component, oper_type, oper_mode,status, initial_size/1024/1024 "INITIAL", target_size/1024/1024 "TARGET", FINAL_SIZE/1024/1024 "FINAL", END_TIME from v$sga_resize_ops order by start_time, component;

With this statement you will see the recent history of resizings.

In this case a slight increase (4 gigs) of the target size of the SGA moved the system away from growing and shrinking the buffers back and forth. And not a single "buffer extermination" was seen afterwards and no peaks in the wait time statistics and the number of resizing ops was down to one per hour. And that was more than okay.

Other solutions would be the deactivation of ASMM (by setting SGA_TARGET to zero) and configuring everything manually(doing it the old way) or setting some reasonable minima for the values controlled by ASMM. Important to know: In the amount specified SGA_TARGET is not only the amount of memory for the four parts mentioned before, it's for the complete SGA. So the amount of memory used for other parts of the SGA than those managed by ASMM has to deducted from the SGA_TARGET size. And this reduced amount of SGA is available for the SGA areas managed by ASMM.

Okay … it's 2012 … and according to some people the world will end this year. However what's really happening? It's the mayan version of the Y2038 problem. While the signed 32 bit integer will send us to the 70ies on 03:14:07 UTC on Tuesday, 19 January 2038, the same happens on 22. December 2012. The mayan calender will send us from 13.0.0.0.0 to 0.0.0.0.0. And sorry, there is more potential in the Y2038 problem to kill us all as in the mayan calendar because nobody used the mayan calendar in embedded systems for nuclear weapons, nuclear power stations, isolation fields for strangelets created in the LHC ( ) , air traffic control. With 32 bit signed integer i'm not so sure

The year almost over. As usual at this time i'm finalizing my tax declaration at the moment. But nevertheless i wanted to write a short article about the last year.

The year started quite bad. In March i had a really bad argument with a good friend and we didn't talked a word since that. Albeit my communication habits could have been better in that days, i assume the writing were on the wall before that day in march and it was just a matter of when, not if, when the argument broke loose.

But starting in June the world just gone downhill for me. It started with the events that finally led to the state the blog currently is. That is really a large thorn sitting in the flesh hurting me all the time. However the consequences were inevitable.

The job had two really great satisfying moments for me with larger successful projects, and a number of others were i was able to help others to find problems. That was really and deeply satisfying moments.

Other things weren't that well as well: Just two main ones … in my vacation i was ill at the weekend and needed almost the complete vacation to get really rid of that nasty bug, and then just 2 days in the job again i broke my ankle and had a torn ankle ligament. While it was painful at least my body seems still to have its good regeneration capabilties in regard of injuries. I don't need painkillers any longer and i'm just wearing the aircast orthotic device at the moment. 24 days is not that bad.

On the good side: My house is almost completed … perhaps 3 month of work in the evening left. So i will have more time to other things soon: Like really pursuing new opportunities - career as well as personal - , finding a new hobby in the year one after almost two years of construction work that rules out any other free time things. Like reading through the heap of books that piled up this year, to get rid of several kilos. Finding a new car in February. Thinking of a nice S60 at the moment. Trying not to break my bones again. And perhaps a longer vacation abroad this year.

I wish you all a good transition into the next year and successful 2012.

Work in Progress - this entry will change often in the next days and weeks

A few days^H^H^H^Hweeks ago, i wrote about simulating the cloud that is most often tagged with the name "network" or "intranet" and sometimes "internet" . This would not be c0t0d0s0.org without an article to explain how you can configure this. This article will explain how you simulate a complete network on a single host with routers, switches, dynamic routing protocols and so on

Scope

At first i want to set the expectations right. I don't want to simulate a cloud in the sense of cloud computing here. I'm thinking about something more complex:



I'm talking about the simulation of this cloud, that often hides a lot of complexities and traps in architectural diagrams.

I wish you all a few quiet days with your loved ones and may all you wishes come true!

Ankle is really hurting. A big thank you to the pharmaceutical industry for providing pain killers. However i would like to concentrate a little bit better again. However things are getting better day by day ...

Just thought, that our perception of the world is heavily influenced by the representations of it ... like a map. Just to demonstrate, a question: Assume you are a pilot. You are flying from Frankfurt to San Francisco. You are a lazy pilot and you are allowed by air traffic control to set the shortest direct course on the autopilot and don't touch it again. What is the initial heading you are setting? Just answer with the first thought! No degrees needed ... just something like west,east,south our southeast.