New series

I’ve started a new weekly series of blog posts called Zen. I’ve spent a lifetime focusing on my work, but never really put other aspects of my life under a magnifying glass before. Hopefully I can change that, and maybe help a person or two.

I’m toying with the idea of making a video/week that showcases my work. I haven’t made the commitment yet, but here’s my first shot, and a glimpse of what that might look like.

Another note on Quadspheres vs QLSC

There is an abundance of posts about Quadrilateralized Spherical Cubes (QLSC) online, but I can’t find any that are sticking to the true nature of Dr. Chan’s original work. It’s a fascinating anomaly that has spread and evolved in the past 6 years, most likely by accident. I contributed to it by making the same exact mistake and had to later publish errata.

Anyway, I love talking about it and the posts seem popular so I would like to share a recent response I made to someone who wrote in. I think it may give just a little bit more detail on what’s going on here. If you want to chime in please feel free to add your comments or write to me too!

Clipped for brevity. Start at the third paragraph for QLSC-only info.

Hello xxxx,

[…]

I’m still using the OpenGL2 pipeline for that project. I initially generate my points using the polar coordinate system. Every point has the same distance from the center of the sphere but different angles. If I wanted more points for more detail, I can average the angles of 2
existing points to add a third point directly between them. Once all those points are generated they are converted to Cartesian (x,y,z) and fed into the vertex buffer to be rendered.

(A side note: You could then animate or alter the terrain of the earth by just changing the distance of each point in the polar coordinate system and that would translate all the way into the final earth mesh!)

I couldn’t find a whole lot of information about Ken’s algorithm either, but I did find is the QLSC algorithm was actually for taking points on a virtual spheroid and making it accurately match up with flat hierarchies of data. I.e., unraveling a sphere into a flat surface to map out some data; and the reverse, taking flat data and mapping it accurately to a sphere. He usually stresses that this is an equal-area mapping algorithm. I don’t really know a whole lot about the specifics here, his papers are hard to read and understand.

He sent me an email after writing his comment, saying it was mostly for stuff like the Cosmic Background Explorer at NASA. They had to study where exactly radiation was coming from and they couldn’t tolerate even small distortions when tracking and mapping this data or it would ruin scientific data crunching. I prodded him for more information but he never got back to me. The guy’s probably super busy, so I’m glad he took the time to message me in the first place.

The lesson I learned is that I wasn’t really using his research at all! Sad, but true. Cubespheres were used well before Ken was born, by cartographers making interesting and wonderful world maps. And that’s what I ended up replicating in my work.

If you look online you will find a lot of other people thought they were using the QLSC algorithm too, but in fact they were just making plain-old cubespheres (or sometimes called quadspheres), and they generally store all their terrain height data in a quadtree data structure, which is better suited for graphics stuff.

I made another article to note my corrections but I left the old one intact because I wanted to keep the comments Ken made. :) I think because so many people were calling this a QLSC–the name really does sound cooler–it caught on quickly and became an artifact of errant
reality. I think it’s neat.

Thanks for writing to me,
-Mark H.

In essence, if you’re looking to do planetary rendering, you’ll want Quadspheres/Cubespheres and Quadtrees specifically; QLSC probably won’t help you. In fact, there are a few solutions out there already, for Ogre or Outerra. UE4 has a wonderful landscape rendering engine that can probably be employed to do planetary rendering if that better suits your needs. UE4 is also pretty.

One last note, NASA also has some really cool work-in-progress for planetary rendering, and a friend has been actively deriving from that work, while another is responsible for the physics simulation behind it. I’m not sure if I can provide any details, but I’m excited about these new advancements in space simulation. I will update this post if I get permission to post snapshots or links and such.

As always, hope this helps.

Slow Performance in Chrome for Ubuntu (Solved)

Google ChromeYou may be experiencing the following while using Chrome under Ubuntu:

  1. Poor system responsiveness
  2. High disk usage
  3. Running out of physical memory, and high swap usage
  4. Corrupted profile when launching Chrome
  5. Crashes from other processes (happens from low memory conditions)
  6. Even after closing Chrome, processes stay in memory and continue to eat up RAM.

Well, the easy fix is pretty simple; just install the beta channel instead!

sudo apt-get install google-chrome-beta

I’m not sure for the reason behind this, but the stable release seems to have memory leak and file I/O issues.

Quadspheres and Spherical Cubes

quadsphereRecently a reader wrote in regarding my earlier Quadrilateralized Spherical Cubes post, however I had neglected to update the original post with some added corrections. I initially wrote about this topic for my first dip into 3D visualization, and unfortunately at the time I conflated QuadSpheres and Quadrilateralized Spherical Cubes as a single concept. Both are a popular topic for visualization, game making and scientific research so I’m going to expand a little bit further on the differences between the two and hopefully that will make it a little bit clearer.

First let me preface this by saying that there’s more than one way to generate a mesh for a sphere. Since my intention was to reduce visual distortion on the earth texture I was using, I chose to map my textures to a QuadSphere, rather than the common method of using ECP tesselation which causes pinching at the north and south poles of the spheroid. A QuadSphere is composed of 6 faces which correlate to the 6 sides of a cube. These faces are called quads because they are essentially squares before you begin any transformations on them. Each of these 6 faces can be subdivided further into smaller quads to increase detail when the camera is closer to the mesh, and each of those resultant quads are tessellated into tris and rendered. In a modern graphics engine this will take place in the vertex shader stage of the rendering pipeline, so you will just feed in the heightmaps and shaders to shape your terrain.

For an older OpenGL 2.0-style pipeline, you can create a QuadSphere by projecting equidistant rays starting from your origin of the sphere to the surface and pushing those vertices into a buffer. This can start out as a 32 x 32 curved grid or something; 1 grid for each side; 6 grids in total to form the spherical shape. You can further subdivide these grids to get more detail when your camera is closer. That’s a QuadSphere in a nutshell.

My work never used the QLSC agorithm. A QLSC ensures that all virtual points on your map take up an equal area of space, decreasing the amount of distortion. Not necessarily texture distortion, but aberration from your virtual model to the real life thing. Distortion occurs naturally when projecting flat data onto a spheroid, so Doctor Kenneth Chan worked extensively on the QLSC algorithm to make it extremely accurate at mapping points and data. This is great for scientific research and that is why it was used in the Cosmic Background Explorer project at NASA.

There are a lot of little caveats to this, but the big takeaway is there’s not a huge need for QLSC in real-time visualization and games. You’ll likely need it for data processing, geomapping spatial data, or other areas of scientific interest.

One last term to take note of is cube mapping. That is how a texture is mapped onto a shape consisting of quads, but not necessarily how that texture will be projected. For instance, skyboxes are meant to be projected on a cube, not a QuadSphere. Though both of these are cube mapped, you will need properly projected textures to suit your needs to minimize the amount of texture distortion.

Have any questions or corrections? Please leave it in the comments below and I’ll be glad to elaborate more.

Pictures of my Arduino

I soldered a couple of 7-segment displays into a protoboard and programmed some look up tables for demos and scrolling text. Nothing fancy, just for fun. I have it running on a loop on my desk now.

Breaking free of the constraints of the Qt data model

I decided to drop the Qt data model altogether in a project. There are certain circumstances that may call for this, and I will outline my reasoning below.

Introduction

Qt has view components that make it easy to tie into a local data model, even from a SQLite source. The widget handles all of the basic functionality for you—for a QListView, this will yield a component that automatically draws, highlights and edits data. This takes a lot of the burden off of you when building a cross-platform product where you’d normally have to manage data, drawing, layout, and native events like flicking to scroll on touch screens. There’s a lot of accessible customization, especially if you’re going the QML route. The drawing is even hardware accelerated where available.

If you want to make a custom component you have to handle each event, drawing and low-level logic for the component manually. You’ll generally need an extraordinary reason to not use the widgets.

Customization

If you have a particularly complex set of data that needs to be displayed there are alternatives to the standard list view; QTableView can display tabular data, QTreeView for hierarchical data sets. For any of these, the items’ paint methods can be overridden to customize the look.

Under nuclear conditions, you can use the non-databound widgets like QListWidget and use its setItemWidget method to convert any list item into a widget of your choosing, including a custom one containing its own child widgets.

The Exceptions

The reasons to not use the Qt model-view ecosystem in a Qt-based application may be pretty obvious in some cases. At some point you’ll cross the very fuzzy threshold of “yes this needs to be completely custom,” and here is a very good combination of reasons why:

  • You have to custom draw every list item,
  • There are multiple or irregularly-shaped hotspots on list items for mouse tracking,
  • There are multiple fields that need to be edited within each list item,
  • There will be a profound number of list items which will trigger many resize events if using custom widgets for your list items,
  • Your data model and view may need to be changed frequently,
  • Your data is asynchronously updated, and you need to ensure scrolling and selection is consistent,
  • Your data is relational or needs to referenced with hashes, and
  • You would have to override a majority of the built-in data model’s methods in order to work with a view anyway.

Qt can of course handle all of these cases so if you’re already neck-deep in using a QListView it wouldn’t make sense to refactor that and write a widget from scratch. However in my case adhering to the Qt convention would take more work for myself, since I need to override all the default functionality anyway.

Ubuntu 14.04 Memory Issues Disappear on Intel CPU

The system I was futzing with last week has an 4-core AMD APU. However I found a desktop with similar specifications, albeit in a 4-core Intel CPU package and it has less trouble with memory management and responsiveness. This could of course be a bug with some module or other hardware, but I’m beginning to notice that low-end laptops that use a combined CPU with GPU package will yield a less responsive experience especially when memory is stressed. Has anyone else noticed this?

10632446_307329906116798_922852833_n

tmux configuration

If you spend about 20 minutes reading the manual and configuring tmux (and tmux sessions for each of your projects) it’ll save you 20 minutes every day that you use the command line.

tmux attach_035

Ubuntu 14.04’s constant memory struggles

Looking at htop I notice that most of my memory is sucked up in graphics-intensive desktop applications (surprise). Here are the big memory killers I’m going to address:

  1. Browser* – 1.2GiB
  2. compiz – 225MiB
  3. X – 175MiB
  4. spotify – 113MiB

*I’ve recently switched to Opera Developer for performance reasons.

4 gibibytes of RAM depleted within an hour of using Ubuntu

The window manager & graphics server take up 400 MiB of RAM combined just by themselves. Reducing the use of windowed applications kills two birds with one stone: less load on the window and composition manager and lower # of SO’s in memory interfacing with X.

The alternative command line work flow

  • tmux – Rather than opening multiple terminal windows. I use Guake as an omni-present terminal.
  • gcalcli – Instead of leaving the Google Calendar tab open. ccal and pal are good alternatives.
  • despotify – Replaces the resource-hungry Linux Spotify client.
  • vim/emacs – RedNotebook is a great lightweight desktop journaling application, but I still find it preferable to maintain a personal journal using vim.
  • mutt with offlineimap – Mutt and offlineimap for email make a great team. Backing up and archiving all your email using this configuration can be automated with one or two line scripts. Side note: Linus himself uses pine for email.
  • bitlbee & weechat – Bitlbee acts as an IRC server, fully managing all of your chat accounts including Google talk (hangouts), Facebook and Twitter. weechat is then used to connect to your bitlbee server. I found this combination yields the smallest returns on performance but it is an important part of my work flow.
  • elinks – Lightweight cli browser.
  • sc – Great for doing simple spreadsheets and calculations without having to load LibreOffice.

Takeaway

There was an order of magnitude gains in memory usage here. I can run all of these at once and the system’s overall response is still much snappier. If you’re comfortable using the command line I will always suggest test-driving CLI alternatives for your favorite applications.

Recovering ecryptfs-encrypted home files from a LUKS encrypted drive in Ubuntu

I did this in Ubuntu 14.04.

This is a bit of a niche situation and it took a really long time so this guide is incomplete. If you run into the situation and need extra help just comment below and I’ll try to give some guidance.

Restoring with rsync

Restoring with rsync

Recently I ran into problems trying to update to the 14.10 release. I lost my video drivers, x11 fallback drivers, and window manager.

Rather than mess with all that I swapped out my laptop’s drive with a fresh one, enabled logical volume management (which the Ubuntu installation disk says is easier to manage) and then installed Ubuntu 14.04 on the new drive. The old one is now a bodged external drive, but it took some finagling to get it to mount.

The first problem I ran into is both of these drives are now boot drives and the logical volume names are both the same. As much as you try, you cannot mount the logical volumes simultaneously without renaming them. Also when you do run sudo lvscan it will show both conflicting logical volume names (ubuntu-vg/root).

I removed my primary boot drive–that’s the important part–then rebooted into a Ubuntu live disc. You should always make a live disk for convenience. Finally I ran lvrename to change the logical volume name of my external. Once I was done with that I rebooted to my primary drive again.

Bingo! Now when I use the disks utility I can mount both logical volumes. You can’t mount it from nautilus, but don’t bother with it. Next is the harder part: You have to mount your home folder that is encrypted with ecryptfs. Look at the post by idallen for a script to mount your private folders at a new mount point. You will have to make modifications to the script. I also manually set the USER variable in the beginning of the script so that I could sudo the script. Make sure you’re pointing to a place on the old drive where your .Private folder is. AND BE CAREFUL WITH THIS. I made the mistake of creating some recursive directory structures in my destination through symbolic links and I could have just as easily deleted the encrypted home folders the same way.

Once the home folder is mounted, you can rsync -r the contents to a new folder. Do not use cp!

That is all. Like I said, if you run into issues let me know in the comments and I’ll try to help.