Odi's astoundingly incomplete notes

Martin Grässlin mentioned last summer that "4.8 we will be able to offer and default to OpenGL ES 2.0 for most of our users". Given the substantial time since that announcement, I wanted to try that with KDE 4.9.3 which was just stabilized in Gentoo today. I tested it on my iMac. Easy enough in Gentoo: enable the gles, gles1, gles2 USE flags and disable opengl for kde-base/kwin. Then recompile Mesa and kwin. Mesa version is 9.0 and xf86-video-ati is at 7.0.0, all on top of a 3.6 kernel.

The result was not too good unfortunately. I quickly realized that all menus were unusable because they either didn't show up at all, or faded in only halfway.  Plus there was massive screen corruption when draging windows around on the desktop.

Same on the Zenbook with Intel SandyBridge and xf86-video-intel-2.20.14.

posted on 2012-11-24 15:30 UTC in Code | 0 comments | permalink

Running a classic setup with Apache as a reverse proxy in front of a Jetty server, I ran into a problem. Apparantly most HTTP threads of Jetty were blocked for long period of times in the following stack within the AJP connector, and no threads were available for the HTTP connector:

   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.ajp.Ajp13Parser.fill(Ajp13Parser.java:203)
	at org.eclipse.jetty.ajp.Ajp13Parser.parseNext(Ajp13Parser.java:265)
	at org.eclipse.jetty.ajp.Ajp13Parser.parseAvailable(Ajp13Parser.java:150)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:662)

The setup is:

• have many Apache threads
• Apache serves some static content and downloads
• Jetty serves the dynamic content
• Jetty has a limited thread pool to accomodate the max allowable load
• the clients are all application clients (not web browsers typically)
• the clients are mostly mobile with more or less shaky over the air connectivity (lots of timeouts, broken connections)

I was a bit surprised that the AJP calls could so easily starve the HTTP calls. Given that HTTP keep-alive and unreliable OTA connections don't mix well, this makes an interesting test.

Test Setup
To find the configuration problem I ran some tests on a local machine:

• 100 clients making a simple dynamic request in parallel
• measure the time to complete all requests

The clients are set to misbehave: after the request is received wait for 60 seconds to close the socket. This is to emulate broken network connections: they never terminate correctly. The clients could be configured to either do HTTP 1.1 keep-alive or not.

Jetty 7.4.3 config:

• QueuedThreadPool.maxThreads = 30
• SelectChannelConnector.maxIdleTime = 300000
• Ajp13SocketConnector.maxIdleTime = 60000

Several Apache configurations were tested:

• Apache 2.2.23
• mod_proxy_http
• mod_proxy_ajp
• differen values for MaxClients

mod_proxy_ajp
ProxyPass / ajp://localhost:8009/ min=1 ttl=20 acquire=10000 timeout=60

MaxClients	keep-alive	Time [s]	timeouts?
150	no	60.1	yes
30	no	11.4	no
150	yes	74.7	yes
30	yes	70.3	yes

mod_proxy_http
ProxyPass / http://localhost:8080/ min=1 ttl=20 acquire=10000 timeout=60

MaxClients	keep-alive	Time [s]	timeouts?
150	no	2.0	no
30	no	6.0	no
150	yes	19.8	no
30	yes	66.0	no

JBoss
For comparison I ran the same test against an old JBoss 4.0.4 which uses Tomcat 5.5.

mod_proxy_ajp
ProxyPass / ajp://localhost:8009/ min=1 ttl=20 acquire=10000 timeout=60

MaxClients	keep-alive	Time [s]	timeouts?
150	no	14.5	no
30	no	9.4	no
150	yes	33.8	no
30	yes	68.5	no

mod_proxy_http
ProxyPass / http://localhost:8080/ min=1 ttl=20 acquire=10000 timeout=60

MaxClients	keep-alive	Time [s]	timeouts?
150	no	2.0	no
30	no	6.0	no
150	yes	21.6	no
30	yes	66.0	no

Conclusions
There is no benefit of using AJP instead of HTTP, the opposite is true. AJP really hurts. I have observed client timeouts only with AJP, never with HTTP. HTTP also was always faster then AJP. The AJP issues are mostly due to a bad implementation in Jetty as (JBoss') Tomcat seems to behave a bit better. But probably mod_proxy_ajp is to blame too, it seems to do something stupid with multiplexing the Apache threads to the AJP backend pool.

This test reveals striking differences between AJP and HTTP and how they behave with connection pools:

• mod_proxy_ajp only works well if the Apache and Jetty connection pools have the same size. A larger the difference make the performance worse.
• mod_proxy_http works better the larger the Apache connection pool is.

This has horrible consequences:
You can not mix HTTP and AJP connectors on the same Apache instance. Because there is no configuration that will work for both. Given the AJP numbers are so much worse than HTTP, and the behaviour with pools is so anti-intuitive, I recommend not to use mod_proxy_ajp at all.
It is advisable to disable keep-alive in scenarios with OTA connections. You can use mod_headers:

RequestHeader set Connection "close"

posted on 2012-10-16 14:29 UTC in Code | 0 comments | permalink

The Radeon board of my old MacBook Pro is dying. During an update the laptop may have overheated. Suddenly the screen got garbled. From now on every few boots only end in a kernel panic (segfault). Compositing refuses to use OpenGL and falls back to using XRender. Also the mouse cursor now has stripes filling the normally empty area of the cursor sprite (see image). The kernel log contains the following messages:

[drm:r100_ring_test] *ERROR* radeon: ring test failed (scratch(0x15E4)=0xCAFEDEAD)
[drm:r100_cp_init] *ERROR* radeon: cp isn't working (-22).

OS-X doesn't want to boot any more. It gets stuck on an empty blue screen. Booting it in safe mode (holding down the shift key) or in single-user mode (holding command-S) works, though. Also there the cursor has the same stripy pattern.
Refit still boots and is usable, although I only get a black screen when it initializes the graphics card in "BIOS mode" most of the time. So I can control Grub only blindly, and I get no early boot messages from the kernel until the KMS driver loads.

Neat workaround for the funny barcode cursor: use the SWCursor option of the radeon driver.

Update: After a few months it got worse. With the radeon driver a lot of horizontal stripes and totally unstable. The machine would hang ever so often. Falling back to the xf86-video-modesetting driver works fine. I still get a stripy mouse cursor but at least it's stable.

Seems it's time to replace this old buddy with a new one soon.

posted on 2012-05-06 16:15 UTC in Code | 3 comments | permalink

If you have wondered how to make a JAX-WS endpoint listen on the wildcard address:

Endpoint.publish("http://0:8080/MyService", new MyService());

Just use 0 as the IP. If you wonder why that works, look at the implementation of Inet4Address.isAnyLocalAddress() which basically checks for zero.

posted on 2012-03-15 15:03 UTC in Code | 0 comments | permalink

When using hostapd on Gentoo, make sure to add the debug USE flag. Otherwise you will never get any error messages from it. It will only exit with error code 1 and no message even if it's just a silly typo in the config file. Also no logging will work without that USE flag.

posted on 2012-03-07 18:43 UTC in Code | 1 comments | permalink

Yey, the 3.3 kernel will report the battery strength of Bluetooth mice and keyboards! The patch just went in.

UPDATE: Now the KDE battery widget can display the capacity of your bluetooth mouse. Just set it to display all batteries.



posted on 2012-01-11 11:58 UTC in Code | 0 comments | permalink

Gentoo lets you define your locale, editor, visual (vim), pager (less). They set the respective environment variables. By default these are unset. Make sure to set them.:

eselect locale
eselect pager
eselect editor
eselect visual
eselect vi

posted on 2011-12-23 13:53 UTC in Code | 0 comments | permalink

I have been running Oracle 11.2 on x86_64 linux with no swap and transparent hugepage support. The system has 6GB RAM and vm.overcommit_memory=0. Oracle is configured to using automatic memory management with MEMORY_TARGET = 4000M. There is nothing else on the system. So 6GB vs 4GB should be plenty, right? Wrong.

While Oracle would run fine most of the time, sometimes connect fails with:

ORA-01034: ORACLE not available
ORA-27102: out of memory
Linux-x86_64 Error: 12: Cannot allocate memory

If you go to the server and check the memory usage, all seems fine however:

~ # free -m
             total       used       free     shared    buffers     cached
Mem:          5972       5714        257          0         11       4078
-/+ buffers/cache:       1624       4347
Swap:            0          0          0

After adding a little swap file (1 GB) the problem goes away. With vm.swappiness=40 the swap usage remains stable at around 250MB. So what's going on here?

Obviously free's output of 1.6GB used memory is wrong. The 4GB SGA lies in the shm filesystem, and obviously doesn't show up here. Why? Because in Linux shm is mapped to the filesystem cache*. I guess this is a trick so that reads from shm do not get copied to cache. So to get the real memory usage you have to add the shm usage to free's output. And that sum is 5.6GB. Which is admittedly a bit much for a 6GB swapless system.

So to fix this, I reduced the SGA to  3.5GB. This results in a bit less than 5GB used memory. And now it works fine without swap!

*) This is a nice example why the linux ate my ram website is not right, when it claims
Disk cache [...] NEVER EVER takes memory away from [applications]. Another example is locked pages in disk cache.

posted on 2011-11-25 13:44 UTC in Code | 2 comments | permalink

Today I recevied an Excel file with dates that look like this:

• 2.Nov.2011 03:00:00 nachm.
• 22.Nov.2011 10:00:00 vorm.

I guess this is an attempt to translate the AM/PM format into German. "nachm." meaning "in the afternoon". However in German the AM/PM notion for time is completely absent. And people always get it wrong (12 AM is noon or midnight?). We use the 24 hour format. Period.

posted on 2011-11-23 11:50 UTC in Code | 0 comments | permalink

Have you ever wondered why the character J appears seemingly randomly in email you receive? Raymond has the explanation.

On a side note this happens for me because my Linux installation does not have the Wingdings font installed.


posted on 2011-10-07 11:07 UTC in Code | 0 comments | permalink