In the world of NoSQL

Googling about couchdb and size limits results in everyone saying that it’s virtually unlimited.  This might be true about couchdb, but not about ext3.  I recently hit the ”max filesize limit” of 2 TB in a couchdb database (luckily just for an internal system).  The result was that couchdb crashed(with the following error) every time the 2 TB database was accessed in any way.

[error] [<0.84.0>] ** Generic server couch_server terminating
** Last message in was {‘EXIT’,<0.416.0>,
{{badmatch,{error,efbig}},
[{couch_db_updater,’-flush_trees/3-fun-0-‘,5},
{couch_key_tree,map_simple,3},
{couch_key_tree,map_simple,3},
{couch_key_tree,map,2},
{couch_db_updater,flush_trees,3},
{couch_db_updater,update_docs_int,5},
{couch_db_updater,handle_info,2},
{gen_server,handle_msg,5}]}}
** When Server state == {server,”/data/couchdb-data”,
{re_pattern,0,0,
<<69,82,67,80,124,0,0,0,16,0,0,0,1,0,0,0,0,0,
0,0,0,0,0,0,48,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,0,93,0,72,25,77,0,0,0,0,0,0,
0,0,0,0,0,0,254,255,255,7,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,77,0,0,0,0,16,171,255,3,0,0,0,
128,254,255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,69,26,84,0,72,0>>},
100,3,”Tue, 13 Sep 2011 13:14:02 GMT”}
** Reason for termination ==
** kill

For me this just meant moving the database from a Ext3 partition on the server to a larger XFS partition instead.  If I would have thought about this up front I would have chosed Ext4 instead of Ext3, which has a limit (depending on how you configure it) of 16 TB.  XFS on the other hand excels with a sizelimit of 8 ExaByte (that’s 1 million terabytes for you that wonders).

Update: The database is still going strong, with 23.5 million documents in it, and currently using 15.4 TB (mostly by attachments)

Read more...

§38 · september 15, 2011 · CouchDB, Filesystem · Kommentarer inaktiverade för Couchdb maximum database size · Tags: , , ,


I recently ran out of disk space on the partition where my couchdb databases resided, the disk had been filled by a couchdb database that severly needed to be compacted (which in my case would reduce it from 270 GB to 40 GB).

The problem was that when couchdb compacts a database, it basically writes a whole new database (so I need 40 GB of free space to be able to perform the compaction).

 

By default a couchdb compaction is performed in three stages:

1. Start/resume compact to dbname.couch.compact

2. Remove dbname.couch

3. Rename(not move) dbname.couch.compact to dbname.couch

 

I couldn’t just create a symlink called dbname.couch.compact to an empty file on another partition, since couchdb removed the file due to invalid format.  It might have worked if I had started a compaction, killed couch, moved dbname.couch.compact to another partition, and then created the symlink, and resumed the compaction, but since that’s not how I did it I don’t know if it will work.

 

My solution was to:

1. Copy dbname.couch to the other partition

2. Stop couch

3. Symlink dbname.couch to dbname.couch on the other partition

4. Start couch and start the compaction.  It should complete the compaction and remove the symlink.

Read more...

§6 · juli 4, 2011 · CouchDB · 1 comment · Tags: ,