Magic Cap Memory

by Dean Yu

The contents of this article have been distilled from a recent presentation. The aim of both the presentation and this article is to dispell some common misconceptions about the Magic Cap memory model, and to describe efficient use of memory in a well behaved Magic Cap package. This article assumes that the reader is already familiar with developing for Magic Cap.

The Basics

Magic Cap has special terms to describe the grouping and relationships between objects. These terms are often the names of Magic Cap classes. The first order of business is to establish the definition of these terms.

A metacluster represents a physical storage device, such as persistent RAM, transient RAM, ROM, and storage cards. There will always be metaclusters for persistent RAM, transient RAM and ROM. A metacluster will be created for a storage card when one is inserted.


A cluster holds a group of related objects. Clusters live inside of metaclusters. Examples of clusters are the system transient cluster, the package uncommitted changes cluster, the package committed changes cluster and the package source cluster.


Most objects are located inside of clusters. A cluster maintains a directory of the objects it contains. This directory is known as the master block. Each object in a cluster has an entry in the cluster's master block, which contains an offset to the location of the object within the cluster. Objects are relocatable blocks of memory; Magic Cap can move objects around inside of clusters to make room for new objects. When the object moves, its master block entry is updated to reflect the new offset to the object. Because objects can move, you almost never use pointers to refer to an object. Instead, you use an ObjectID, which describes the cluster an object is in, and the master block entry containing the offset to that object.

A context tracks all clusters related to a particular package, or to the system. The system has a context object, called the system context, which groups together the clusters belonging to the system. Each package has a context object, called the package context, which groups together the clusters used by the package. Every package context also references the clusters belonging to the system. This means that system objects are always accessible from any package. Objects from one package are not immediately accessible from another package. Context objects live in the persistent RAM metacluster.


Context objects store the clusters and metaclusters associated with it in their extra data. Each cluster slot in a context object's extra data contains two entries. The first entry is the ObjectID of a cluster. The second entry is a pointer to the beginning of the cluster object.

cluster number cluster slot contents cluster slot contents cluster number
cluster 8 A0000004 00BB5118 A0000001 00B5A190 cluster 88
cluster 9 00000000 00000000 00000000 00000000 cluster 98
cluster A 40B5A134 00B5A140 40AED03C 00AED048 cluster A8
cluster B A800000B 00B252DC 00000000 00000000 cluster B8
cluster C A800000C 00B25AE8 00000000 00000000 cluster C8
cluster D A8000002 00B34A2C 00000000 00000000 cluster D8
cluster E 00000000 00000000 A8000001 00AED098 cluster E8
cluster F 00000000 00000000 00000000 00000000 cluster F8
cluster 10 40714B82 00714B8E A0000003 00B8310C cluster 11
cluster 12 88000001 00B5A5F4 A0000017 00B5C8F8 cluster 13

The con command in the debugger will show you this information about the current context. To dump the clusters associated with another context, you need to first dump the context object, then dump the extra data of the context object:

dobj <context>
dm . + <Context_dataOffset> $a0

You can tell the cluster an object resides in by looking at the high nybble of the object's ObjectID. Clusters 10, 11, 12, and 13, which represent the source and uncommitted changes clusters for the system and the package, are considered non-addressible. That is to say, you cannot create an ObjectID that will directly reference an object in any of these clusters. How these objects are accessed is discussed in the next section.

Creating Objects

Persistent objects can be in one of three clusters: the uncommitted changes cluster, the committed changes cluster, or the source cluster. In order to preserve the consistency of persistent data, Magic Cap first creates persistent objects in uncommitted changes clusters. Periodically, Magic Cap will move the objects in the uncommitted changes cluster into the committed changes cluster.


Even though uncommitted changes clusters are referred to as persistent clusters, they are actually located in transient memory. The system uncommitted changes cluster is known as cluster 8. The package uncommitted changes cluster is known as cluster B. When you specify a persistent object, Magic Cap first looks in the uncommitted changes cluster for the object. If the object doesn't exist in that cluster, Magic Cap looks in the committed changes cluster for the specified object. If the object doesn't exist there, either, Magic Cap looks in the source cluster. Thus, the high nybble of the ObjectID for a persistent object represents the first cluster of a cluster chain Magic Cap will search for that object.

There are no transient uncommitted clusters. This means that accessing transient objects is faster than accessing persistent objects, because Magic Cap does not have to search additional clusters to find the requested object.

The MemoryMonger package provides a working demonstration of how creating new objects affects the amount of memory available in transient and persistent clusters. Play with MemoryMonger and notice how persistent objects first take up space in the uncommitted changes cluster, and how the free space changes when those objects are moved into the committed changes cluster. Also watch how memory becomes available when you destroy objects.

When a package creates an instance of a class imported from another package, several objects are created in memory. The first object is created when the package calls Import() to import a class number from another package. The object returned from the Import() call is a Reference object located in the system context. When the foreign class is actually instantiated, two new objects are created in the package context. The first object is an instance of the ClassImport class. This is a special type of class that refers back to the actual class in the other package. This ClassImport object is added to the importing package's class list and given a unique class number. The second object is the instance itself, but its class number is the class number that was newly assigned to the ClassImport object, not the class number imported from the other package.


Changing Objects

Magic Cap never modifies a committed persistent object in place. Instead, the object is shadowed into the uncommitted changes cluster. Magic Cap does the same thing when code changes an object in the source cluster. This means that whenever you change an object, you are essentially creating a new object in the uncommitted changes cluster. If the object being modified exists in the uncommitted changes cluster, the modifications are made in place. When Magic Cap performs its periodic updating of the committed changes cluster, the changed objects are moved from the uncommitted changes cluster, committing the changes.

The calls BeginModifyFieldsOf() and BeginModifyExtra() will create shadow objects and return pointers to the newly created uncommitted shadows. This is because Magic Cap tries to protect the data in committed changes clusters, so you will not get writeable pointers to objects in these clusters.

Because transient clusters are not shadowed, changing transient objects is more efficient than changing perstent objects.

Destroying Objects

When Destroy() is called to free the memory occupied by an object, be aware that the memory occupied by persistent objects may not be reclaimed immediately. Because object destruction is considered a change to the object, Magic Cap will not immediately destroy objects in the committed changes or source clusters. Instead, a master block entry will be created for this object in the uncommitted changes cluster that marks that this object is scheduled for destruction. The memory in the occupied by an object in the committed changes cluster will be freed the next time changes are committed. If this object is requested before the commit occurs, Magic Cap will return that the object does not exist because it is marked as "destroyed" in the uncommitted changes cluster's master block. Calling Destroy() on an object that has not be committed yet, i.e., it's still located in the uncommitted changes cluster, will immediately free the memory occupied by the uncommitted object. Because transient clusters are not shadowed, destroying an object in any transient cluster will also make the memory occupied by that object available immediately.

Most objects maintain strong relationships with other objects. This means that if one object refers to a second object, when the first object is destroyed or copied, the second object will be destroyed or copied with it. You can define a weak relationship between objects by specifying the noCopy keyword on field declarations in class definitions. (Newer versions of the class compiler allows the use of weak as a synonym for noCopy.) When an object weakly refers to a second object, the second object is not destroyed or copied with the first object.

Classes can specify that their instances are shared. Shared objects are not destroyed when objects that refer to them are destroyed. However, calling Destroy() directly on a shared object will destroy the object. Indexicals behave similarly to shared objects during object destruction. If the field of an object being destroyed contains an indexical reference, the object referred to by the indexical will not be destroyed. However, calling Destroy() directly on an indexical value will destroy the object the indexical refers to.


How Magic Cap Allocates Memory

Magic Cap goes through five phases when trying to create a new object: request, throw up, purge, compact, and grow.

In the request phase, Magic Cap simply looks at each free block in a cluster to see if there is one free block that is large enough to create the new object from. If such a free block is found, the new object is chopped off the block, and a master block entry is allocated for the new object.

If there is no single free block in the cluster large enough to create the new object from, Magic Cap relocates objects to move free blocks next to each other so that they can be merged. This is the throw up phase. Magic Cap can't move blocks around locked blocks. This is why we always tell you not to change objects or allocate memory while an object is accessed. An object allocation request that encounters a locked object in the cluster will fail more often than not. Objects that will be permanently locked should be created in one of the locked clusters. These clusters are located at the bottom of their respective metaclusters so that they don't get in the way of object allocation/relocation inside the metacluster.

After all free blocks have been merged, and there is still not enough space in the cluster to allocate the new object, Magic Cap will start purging existing objects. An object passes through five purging levels before Magic Cap actually destroys it. This corresponds roughly to aging objects before they are destroyed, so older objects are purged before newer objects. After each object is purged, Magic Cap checks to see if enough memory is free to fulfill the allocation request. If there is, purging stops, and Magic Cap restarts the request phase to merge the new free blocks into the existing free block.

If, after all purgeable objects have been destroyed, there is still not enough memory to fulfill the allocation request, Magic Cap will try to compact objects. This is done by calling Compact() on objects, telling them to shrink themselves. Typically, the only objects that can shrink themselves in Magic Cap are list objects. If you create a class that can shrink its instances, you should override Compact().

In the last phase, Magic Cap will try to grow the cluster to make more free space. This is done at the expense of compacting other clusters in the same metacluster. If this phase fails to create enough free space to fulfill the allocation request, Magic Cap throws a cannotAllocateMemory exception. If this exception is uncaught, Magic Cap will reset if transient memory was full. If persistent memory was full and the exception is uncaught, Magic Cap will bring up the out of memory window.


Magic Cap performs garbage collection to reclaim persistent memory used by objects that are not referenced from other objects. This garbage collection is performed when Magic Cap communicators are powered off, and before the out of memory window is shown. Garbage collection is not performed on transient objects. Furthermore, transient objects are not scanned to see if they refer to persistent objects. This means that a persistent object that is only referenced from a transient object will be garbage collected anyway. Don't keep references to persistent objects in transient objects.

A lightweight form of garbage collection is performed every few minutes. During this time, string dictionaries which contain names of objects are stabilized, and entries for deleted objects are removed.

Reference Objects

Reference objects are used to access objects within a package context from outside of that context. Because persistent objects from different packages share the same ObjectID space, the Reference object is needed to specify which package context the object lives in. Since an ObjectID is enough to uniquely identify an object in a system cluster, Reference objects are created there. This is the reason why most users see the amount of built-in memory in their communicators go down with prolonged usage. Since Reference objects are created automatically when they're needed, and packages almost never care that a Reference object exists, this is overhead for using Magic Cap.

Reference objects are created whenever a package object is referenced from outside of that package's context. This happens when another package calls Import() to import an object or class number from the package, or when the package calls an operation defined by another package and passes package objects as parameters.

Reference objects are destroyed when the package object it references is destroyed. Reference objects are also destroyed when the package that owns the object being referenced is packed up. Persistent Reference objects that are not referenced by other objects can be garbage collected. Transient Reference objects are also destroyed when Magic Cap resets.


Cleaning is when Magic Cap sets the value of a field to nilObject because the object that was contained in that field is no longer valid. Cleaning happens when a package is about to go away. Fields that are cleaned contain references to objects in another context. When a package is packed up, fields of objects that contain references to objects in another package context (i.e., the field contains a Reference object), or to objects in the system context are set to nilObject. References in other contexts to objects in the package being packed up are also set to nilObject. Fields that contain an indexical value are not cleaned.

The important thing to realize is that fields that have been cleaned are not filled in with the values they used to contain when the package context that went away comes back. The only way that Magic Cap has of hooking objects together is with the install/receiver mechanism. Objects that are not in these lists cannot be hooked back together. Because most cross-context references do not appear in these lists, Magic Cap cannot piece them back together. This is also why it's important to keep subviews of a fileable viewable in the same cluster as the viewable. Consider the case where a card is created using a Preferred memory call; it might wind up in the new items package, which is a different package context. If the subviews of this card are not created in the same cluster as the card, you might wind up with a cross-context view chain. If the software package and the new items package are ever separated (one is packed up, or the two live on different physical storage devices and one is removed), the view chain will be broken and will not be pieced together.

Preferred Memory Calls vs. Near Memory Calls

There are two families of calls that the majority of objects should be created with. The Preferred memory calls, NewPreferred(), CopyPreferred() and MovePreferred(), will create objects based on where the user has new items set to go. The Near memory calls, NewNear(), CopyNear() and MoveNear() will create objects in the cluster where the object specified by the nearThis parameter is located. You can use special constants that represent specific clusters to create a new object in a particular cluster with the Near memory calls.

You should almost always create objects using the Near memory calls. You would only use the Preferred memory calls to create viewable objects that can be filed, like cards and tasks. To create a view chain in the preferred container, use a Preferred memory call to create the root object, then use Near memory calls to create subview objects in the same cluster as the root viewable. Using the Near memory calls is always preferred!

Objects can be created in different clusters depending on whether the object is an instance of a system class or package class, and depending on the location of the package creating the object. The following table shows which cluster an object will be created in depending on the type of memory call used and where the calling package is located.

Type of call used Package in ROM Package in main memory Package on RAM card
system class, near package
cluster b,
main memory
cluster b,
main memory
cluster b,
card memory
package class, near package
cluster b,
main memory
cluster b,
main memory
cluster b,
card memory
system class, main memory
cluster 8,
main memory
cluster b,
main memory
cluster 8,
main memory
package class, main memory
cluster b,
main memory
cluster b,
main memory
cluster b,
card memory
system class, card memory
new items,
card memory,
reference in cluster 8
new items,
card memory,
reference in cluster 8
new items,
card memory,
reference in cluster 8
package class, card memory
new items,
card memory,
reference in cluster 8
new items,
card memory,
reference in cluster 8
new items,
card memory,
reference in cluster 8

Even though the table uses cluster 8 and cluster b to describe the locations of persistent objects, keep in mind that the uncommitted changes clusters live in transient memory, which comes from built-in memory. The committed changes cluster will be located on the physical storage device described in each table entry.

Transient Memory vs. Persistent Memory

There are two types of memory in Magic Cap. The contents of persistent memory are preserved when Magic Cap resets or the communicator is turned off. Persistent memory is where user data is stored. The contents of transient memory are lost when Magic Cap resets. In Magic Cap 1.0, the contents of transient memory are also lost when Magic Cap communicators are powered off. (Magic Cap 1.5 introduced the "instant on" feature which preserves the contents of transient memory across power downs.)

Transient memory should be used to hold temporary objects, or for objects that can be rebuilt from other data. The most common cause of resetting communicators is when there is no more transient memory available for use. While there is only a limited amount of transient memory available, this does not mean you should avoid using transient memory completely. Transient memory is more efficient than persistent memory because it is not shadowed. By following some basic rules about how transient memory should be used, more transient memory would be available for everyone to use.

Don't be a hog

Transient memory is typically used for objects that are short lived. Because transient memory is a scarce resource, you should try to allocate transient memory only when you need it. Some packages reserve a chunk of transient memory at install time that their package uses later. The problem with this approach is that this reduces the amount of transient memory that's available to the system and other packages that might have more immediate transient memory needs. Allocate from transient memory when you're actually going to use it; don't hold on to a chunk for later use. Get rid of your transient objects once you're done with them. By not being a hog, more transient memory is made available for everyone.

Be prepared for failure

How many times have we all seen the "your communicator is running low on memory" message after a clean up? This message comes up because some code somewhere assumed that it could always get the transient memory it needed. It didn't set up an exception handler, so when Magic Cap couldn't fulfill the allocation request, the resulting cannotAllocateMemory exception got thrown all the way up to the root handler that forces Magic Cap to reset. The single most important thing you can do in your package to prevent clean ups from happening is to catch cannotAllocateMemory exceptions.

This doesn't mean that you need to set up an exception handler every single time. To be honest, if you can't allocate a 32-byte transient object for the duration of one routine, things are in pretty bad shape. But if you're going to be storing references to a transient object in some other object, or the transient object is relatively large (more than a few hundred bytes), you should be prepared to catch cannotAllocateMemory exceptions. Every out of memory exception that is caught makes the Magic Cap experience that much better.

Don't blindly allocate transient memory in Reset()

In Magic Cap 1.0, Load(), Install(), and Reset() are called whenever transient clusters need to be rebuilt. This means that these operations are called whenever the communicator is powered on. Install() and Reset() are both class operations, so were sometimes used interchangeably.

Transient clusters are not destroyed in Magic Cap 1.5 when power is restored to the communicator. This means that Load() and Install() are not necessarily called in these cases any more. However, Reset() is still called on every power on to notify classes of the event. This means that Reset() methods that blindly allocate transient memory because they assume that transient clusters have been recreated are actually leaking transient memory. These methods should be re-written to either check to see if the transient object needs to be created, or move the allocation be performed into the Install() method.

Locked Objects

Objects can normally be moved around within clusters to make room for new objects. However, certain calls will lock an object in place, prevent it from moving. Because Magic Cap cannot move other objects around locked objects, you should only keep objects locked for short periods of time. You should never try to allocate memory while an object is locked. Because any Magic Cap operation can potentially allocate memory, you should avoid calling other operations while objects are locked. Keep in mind that committing changes and shadowing can cause memory to move as well.

The right way to use BeginModify()

This is the correct way to use a call that locks an object. The object is kept locked for a short duration, and no Magic Cap operations are called while the object is locked.

newCard = CreateNewCardNear(stationery, stack);
objectPtr = BeginModify(object);
   objectPtr->card = newCard;

The wrong way to use BeginModify()

This is an incorrect use of a call that locks an object. While the object is kept locked for a short duration, a Magic Cap operation that allocates memory is called while the object is locked. This will fail more often than not.

objectPtr = BeginModify();
   objectPtr->card = CreateNewCardNear(stationery, stack);

Calls that return pointers to objects will cause those objects to be locked. These calls are BeginReadFieldsOf(), BeginModifyFieldsOf(), BeginReadExtra() and BeginModifyExtra(). The corresponding End call will normally unlock the object. Magic Cap tracks accesses to locked objects, so that a call to EndRead() will not unlock an object if BeginReadFieldsOf() has been called on the object twice. Because BeginModifyFieldsOf() and BeginModifyExtra() will cause a shadow object to be created in the uncommitted changes cluster, you should never nest Modify calls.

Two final calls, NewLockedBuffer() and NewTransientBuffer(), will create locked objects in the locked transient cluster.

Purgeable Objects

Magic Cap will purge objects from clusters to make room for new objects. Objects are not purgeable by default; you need to call the SetPurgeable() method to make an object purgeable. Making objects purgeable helps Magic Cap to reclaim memory for other uses. Candidates for objects that can be purgeable are objects that can be rebuilt from other data. You should also consider making large transient objects purgeable as well.

Purgeable objects are destroyed automatically if Magic Cap needs to reclaim the memory. You can override Purge() to be notified when an object is about to be purged.

Shared Objects

Magic Cap shares objects on a class by class basis. That is to say, all instances of a class that returns true from IsShared() will be shared. Typically, shared objects are objects that are large, and copying them would waste memory. Examples of shared objects are sounds, images and address cards.

In the optimal situation, there should be one unique instance of a shared object in any given cluster. If two stamps look the same, there should only be one Image object which is referenced by both Stamp objects. To ensure this optimal case, you should call DeleteDuplicate() every time you change or create a shared object. DeleteDuplicate() searches for objects that have the same checksum as your object in the cluster. If one is found, your instance is destroyed, and the ObjectID of the existing object is returned to the caller of DeleteDuplicate(). Any code that modifies a shared object but fails to call DeleteDuplicate() is potentially creating an extra copy of a shared object.

Using DeleteDuplicate()

This code snippet will cause a new LineStyle object to be created in the same cluster as the object referred to by self. Because line styles are shared objects, DeleteDuplicate() is called in case the new object matches an existing line style object in this cluster.

if (HasObject(lineStyle))
   lineStyle = CopyNear(lineStyle, self);
   lineStyle = NewNear(LineStyle_, self, nil);
SetLineStyle(self, DeleteDuplicate(lineStyle));

Because a shared object can be referenced by more than one other object, you should not modify a shared object directly. Instead, make a copy with CopyNear(), make your changes, then call DeleteDuplicate() on your modified copy.

Object Ownership

One of the reasons memory leaks occur in Magic Cap is that it is often unclear who is responsible for the destruction of a given object. This last section presents object ownership conventions that helps to define the relationships between objects. While not all areas of Magic Cap itself follow these conventions, doing so in your own package will help you isolate where memory leaks are occuring.

The key to these conventions is the encapsulation of data in an object. You should consider the data fields in objects to be private to that object; use the public attributes to access data from an object. In your own classes, you should define your getter attributes to return transient copies of the requested data; never return an object stored in a field. It is the responsibility of the caller of your getter to move the copy to where it wants it, and to destroy the copy when the caller is done with the data. Similarly, your setter attributes should never store the parameter that is passed in. Instead, it should call CopyNear() to move the parameter object into the same cluster as the object that will refer to it. This way, your setter does not make assumptions about what cluster the parameter object is located in.

This means that auto-getters and auto-setters will be less useful, since these do not do the appropriate copying. You can still use these for non-object field data, but you should write custom getters and setters for attributes that work with objects.

Remember All That?

After reading this article, you should have a clearer understanding of how the Magic Cap memory system works, and where memory goes when you create objects. You should also be well armed to side step many of the land mines that are associated with creating stable Magic Cap packages. As more packages follow the rules and conventions established here, Magic Cap itself should seem more stable to users over time.

Tech Docs Magic Cap in Depth