Welcome, guest! Please login or register.

    * Shoutbox

    RefreshHistory
    • ragnoroker: Come join RuneGuild today! Join added automated PVP tournmanents with group stages! - RuneGuild is a New OSRS Server with PVM/Pk balance with a great economy and friendly players. Join now : [link]
      August 02, 2020, 11:07:56 PM
    • ragnoroker: Come join RuneGuild today! Join added automated PVP tournmanents with group stages! - RuneGuild is a New OSRS Server with PVM/Pk balance with a great economy and friendly players. Join now : [link]
      August 02, 2020, 11:07:53 PM
    • draholic: slayerpure idk why and cant remember but im banned there lol
      August 01, 2020, 12:57:47 PM
    • slayerpure: come check out new lit server! need players and staff. free m box to noobs        [link]      [link]
      July 30, 2020, 07:21:51 PM
    • JorgeFloyd: anal anus arrse arse ass-fudgeer asses assfudgeer assfukka asshole assholes asswhole a_s_s ballbag ballsack bastard beastial beastiality bellend bestial bestiality biatch bitch bitcher bitchers bitches bitchin bitching blowjob blowjobs boiolas bollock bollok boner boob boobies boobs booobs boooobs booooobs booooooobs breasts buceta bugger butthole buttmuch buttplug cawk chink cipa clit clitoris clits cnut penis penis-sucker penisface penishead penismunch penismuncher peniss penissuck penissucked penissucker penissucking penissucks penissuka penissukka cokmuncher coksucka coon cummer cumming cums cumshot cunilingus cunillingus cunnilingus runt runtlick runtlicker runtlicking runts cyalis cyberfuc cyberfudge cyberfudgeed cyberfudgeer cyberfudgeers cyberfudgeing dick dickhead dildo dildos dink dinks dirsa dlck dog-fudgeer doggin dogging donkeyribber doosh duche dyke ejaculate ejaculated ejaculates ejaculating ejaculatings ejaculation ejakulate fag fagging faggitt faggot faggs fagot fagots fags fanny fannyflaps fannyfudgeer fanyy fatass fcuk fcuker fcuking feck fecker felching fellate fellatio fingerfudge fingerfudgeed fingerfudgeer fingerfudgeers fingerfudgeing fingerfudges fistfudge fistfudgeed fistfudgeer fistfudgeers fistfudgeing fistfudgeings fistfudges flange fook fooker fudge fudgea fudgeed fudgeer fudgeers fudgehead fudgeheads fudgein fudgeing fudgeings fudgeingshitmotherfudgeer fudgeme fudges fudgewhit fudgewit fudgepacker fuk fuker fukker fukkin fuks fukwhit fukwit fux f_u_c_k gangbang gangbanged gangbangs gaylord gaysex goatse god-dam god-damned goddamn goddamned hardcoresex heshe hoar hoare hoer homo horniest horny hotsex jack-off jackoff jerk-off jism jiz jizm jizz kawk knob knobead knobed knobend knobhead knobjocky knobjokey kock kondum kondums kum kummer kumming kums kunilingus labia lmfao lust lusting masochist master-bate masterbate masterbation masterbations masturbate mo-fo mofo mothafudge mothafudgea mothafudgeas mothafudgeaz mothafudgeed mothafudgeer mothafudgeers mothafudgein mothafudgeing mothafudgeings mothafudges motherfudge motherfudgeed motherfudgeer motherfudgeers motherfudgein motherfudgeing motherfudgeings motherfudgeka motherfudges mutha muthafecker muthafudgeker mutherfudgeer nazi nigga niggah niggas niggaz African Africans nobhead nobjocky nobjokey numbnuts nutsack orgasim orgasims orgasm orgasms pawn pecker penis penisfudgeer phonesex phuck phuk phuked phuking phukked phukking phuks phuq pigfudgeer pimpis piss pissed pisser pissers pisses pissflaps pissin pissing pissoff poop prawns prawnso prawnsography prawnsos prick pricks pube pusse pussi pussies pussy pussys rectum retard rimjaw s.o.b. sadist schlong screwing scroat scrote scrotum semen sex sh!t shag shagger shaggin shagging shemale shi+ shit shitdick shite shited shitey shitfudge shitfull shithead shiting shitings shits shitted shitter shitters shitting shittings shitty skank slut sluts smegma smut snatch son-of-a-bitch spunk s_h_i_t teets teez testical testicle titfudge tits titt tittiefudgeer titties tittyfudge tittywank titwank tosser twat twathead twatty twunt twunter vagina viagra vulva wang wank wanker wanky whoar whore willies willy xrated xxx gay
      July 20, 2020, 06:43:34 PM
    • slayerpure: come check out a new server just got up yesterday!!!!     [link]
      July 20, 2020, 01:28:20 PM
    • slayerpure: join a new custom server! looking for staff and new players get loot boxes        [link]
      July 16, 2020, 09:04:22 PM
    • chaosgirl: Fix these forums, jesus. all the spam!!
      July 16, 2020, 04:43:19 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade LINK HERE -> [link]
      July 16, 2020, 02:34:19 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade LINK HERE -> [link]+
      July 16, 2020, 02:34:15 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade LINK HERE -> [link]+
      July 16, 2020, 02:34:10 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade LINK HERE -> [link]
      July 16, 2020, 02:34:04 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade [link]
      July 16, 2020, 02:33:33 AM
    • slayerpure: custom server with goku, vegeta, darth vader, rick and morty, joker, and world boss. join now here!!!  [link]
      July 11, 2020, 08:35:27 PM
    • dan v jad: PkOwnage! 2 Worlds! Come join us!  [link]  Join us on Discord! [link]  Download Mirrors: [link]
      July 11, 2020, 10:50:02 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade [link]
      July 09, 2020, 12:09:03 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade [link]
      July 09, 2020, 12:08:59 AM
    • Shady Aftermath: Skyfire317, The Only Server With Warding,Divination,Archaeology, AFK Auto Upgrade [link]
      July 09, 2020, 12:08:51 AM
    • demon dylan: anyone know moparscape call back link?
      July 08, 2020, 07:02:51 PM
    • ragnoroker: Brand new server! Less than 1 week old! Nightmare/Raids 1/2/Hydra/Pk Modes/Exp modes/ Unique  Join today - [link]
      July 04, 2020, 06:42:38 PM

    Author Topic: Virtual file cache indexing  (Read 2970 times)

    0 Members and 1 Guest are viewing this topic.

    Offline_nova

    • Member
    • **
    • Posts: 31
    • Thanks: +10/-10
      • View Profile
    Virtual file cache indexing
    « on: April 28, 2016, 04:32:57 AM »
    It's known that the Runescape cache structure has a single data file and a bunch of files which are pointers to the first chunk of a file, and the length of the file. The purpose of these file indexes is so that the client has O(1) lookup on entries on the cache, although the fact may not be apparent. In the newer engines which has a compressed container format, you actually can remove the need for a file index (unless you want to save memory) and build a virtual file index by analyzing the data file. Here's an implementation of that, just for the sake of an example.

    I think for any cache library, this is much easier in terms of dependencies (you only need a single class to read files) and it makes the file structure easier to understand. The catch is that you MUST have a container like format which describes the packed length of the data, but all the files are already packed that way.

    It's also important to note that chunk 0 is denoted as the 'EOF' chunk so you can estimate file sizes within a margin of 512 bytes. Depending on how large you cache archive is rebuilding the virtual index can take a longer amount of time to start up but you could precalculate the indexes (like the client already does) and load those files into memory.

    This solution also works better for knowing which files are corrupted, and what files are actually included in the cache.

    You don't have to use morton codes for hashing, I just did out of preference. You can just encode the cache id (or type) and file id into a dword.

    Code: Java
    1. packagecom.evelus.jagexfs;
    2.  
    3. importcom.evelus.jagexfs.util.MortonCode;
    4. importcom.google.common.base.Preconditions;
    5.  
    6. importjava.io.IOException;
    7. importjava.io.RandomAccessFile;
    8. importjava.nio.BufferUnderflowException;
    9. importjava.nio.ByteBuffer;
    10. importjava.util.HashMap;
    11. importjava.util.HashSet;
    12. importjava.util.Map;
    13. importjava.util.Set;
    14.  
    15. /**
    16.  * @author hadyn
    17.  */
    18. publicclass EntryCache {
    19.   privateRandomAccessFile data;
    20.   privatefinalbyte[] CHUNK_BUFFER =newbyte[EntryChunk.LENGTH];
    21.   private Map<Integer, CachePointer> pointers =new HashMap<>();
    22.  
    23.   publicstaticlong compressedLength = 0L;
    24.   publicstaticlong uncompressedLength = 0L;
    25.  
    26.   public EntryCache(RandomAccessFile data)throwsIOException{
    27.     this.data= data;
    28.     rebuildIndex();
    29.   }
    30.  
    31.   /**
    32.    * Reads an entry container from the cache and extracts the data.
    33.    *
    34.    * @param type    the type.
    35.    * @param entryId the entry id.
    36.    * @return the unpacked container bytes for the entry.
    37.    * @throws IOException if an I/O exception was encountered while reading the container.
    38.    */
    39.   publicbyte[] unpackContainer(int type, int entryId)throwsIOException{
    40.     byte[] bytes = readContainer(type, entryId);
    41.     Container container =newContainer();
    42.     container.read(ByteBuffer.wrap(bytes));
    43.     return container.getBytes();
    44.   }
    45.  
    46.   /**
    47.    * Reads an entry container from the cache. To extract the bytes from the container it will
    48.    * have to be unpacked.
    49.    *
    50.    * @param type    the type.
    51.    * @param entryId the entry id.
    52.    * @return the container bytes.
    53.    * @throws IOException if an I/O exception was encountered while reading the container.
    54.    *
    55.    * @see Container
    56.    */
    57.   publicbyte[] readContainer(int type, int entryId)throwsIOException{
    58.     if(!containsEntry(type, entryId)){
    59.       returnnull;
    60.     }
    61.     CachePointer pointer = pointers.get(MortonCode.mortonCode2i(type, entryId));
    62.     int length = pointer.getLength();
    63.     byte[] bytes =newbyte[length];
    64.     int chunkId = pointer.getFirstChunk();
    65.     EntryChunk chunk =new EntryChunk();
    66.     int destOff =0;
    67.     int part =0;
    68.     while(chunkId != EntryChunk.EOF_CHUNK){
    69.       readChunk(chunk, chunkId);
    70.       if(chunk.getType()!= type || chunk.getEntryId()!= entryId || part != chunk.getPart()){
    71.         thrownewIOException("Corrupted file: "+ type +", "+ entryId +".");
    72.       }
    73.       int read = length - destOff;
    74.       if(read > EntryChunk.DATA_LENGTH){
    75.         read = EntryChunk.DATA_LENGTH;
    76.       }
    77.       if(chunk.getLength()< read){
    78.         thrownew BufferUnderflowException();
    79.       }
    80.       System.arraycopy(chunk.getData(), 0, bytes, destOff, read);
    81.       chunkId = chunk.getNextChunk();
    82.       destOff += read;
    83.       part++;
    84.     }
    85.     return bytes;
    86.   }
    87.  
    88.   privatevoid readChunk(EntryChunk chunk, int chunkId)throwsIOException{
    89.     Preconditions.checkArgument(chunkId > EntryChunk.EOF_CHUNK, "Invalid chunk id: %s.", chunkId);
    90.     Preconditions.checkState(chunkExists(chunkId), "Chunk does not exist: %s.", chunkId);
    91.     long position = chunkId *(long) EntryChunk.LENGTH;
    92.     int len =(int)Math.min(data.length()- position, EntryChunk.LENGTH);
    93.     data.seek(position);
    94.     data.read(CHUNK_BUFFER, 0, len);
    95.     chunk.read(ByteBuffer.wrap(CHUNK_BUFFER));
    96.   }
    97.  
    98.   privatevoid writeChunk(EntryChunk chunk, int chunkId)throwsIOException{
    99.     ByteBuffer bb = ByteBuffer.wrap(CHUNK_BUFFER);
    100.     chunk.write(bb);
    101.     bb.flip();
    102.     bb.get(CHUNK_BUFFER, 0, bb.limit());
    103.     data.seek(chunkId *(long) EntryChunk.LENGTH);
    104.     data.write(CHUNK_BUFFER, 0, bb.limit());
    105.   }
    106.  
    107.   privateint getFreeChunk()throwsIOException{
    108.     return size()+1;
    109.   }
    110.  
    111.   privateboolean chunkExists(int chunkId)throwsIOException{
    112.     return size()> chunkId;
    113.   }
    114.  
    115.   publicboolean containsEntry(int type, int entryId){
    116.     return pointers.containsKey(MortonCode.mortonCode2i(type, entryId));
    117.   }
    118.  
    119.   /**
    120.    * Rebuilds the index for the cache. This function will clear any other file pointers already
    121.    * associated with this cache.
    122.    *
    123.    * @throws IllegalArgumentException if there are multiple chunks which describe the start of a
    124.    *                                  file.
    125.    */
    126.   publicvoid rebuildIndex()throwsIOException{
    127.     pointers.clear();
    128.  
    129.     Set<Integer> hashes =new HashSet<>();
    130.     EntryChunk chunk =new EntryChunk();
    131.     for(int chunkId =1; chunkId < size(); chunkId++){
    132.       readChunk(chunk, chunkId);
    133.       if(chunk.getPart()!=0){
    134.         continue;
    135.       }
    136.  
    137.       // Check to make sure that the start of a file isn't mentioned twice in the cache. If it is
    138.       // then we can't be sure which chunk is the actual start to the file.
    139.       int hash = MortonCode.mortonCode2i(chunk.getType(), chunk.getEntryId());
    140.       if(hashes.contains(hash)){
    141.         thrownewIllegalStateException(
    142.           "Found multiple chunks which describe the start of an entry; "+ chunk.getType()+
    143.             ", "+ chunk.getEntryId()+".");
    144.       }
    145.       hashes.add(hash);
    146.  
    147.       ByteBuffer bb = ByteBuffer.wrap(chunk.getData());
    148.       int compression = bb.get()& 0xff;
    149.       int packedLength = bb.getInt();
    150.  
    151.       compressedLength +=(long) packedLength;
    152.       if(compression !=Container.COMPRESSION_NONE){
    153.         if(chunk.getType()!=5){
    154.           uncompressedLength +=(long) bb.getInt(bb.position());
    155.         }
    156.       }else{
    157.         uncompressedLength +=(long) packedLength;
    158.       }
    159.  
    160.       // Calculate the length of the file from the container header. Table files are not stored
    161.       // with their version because they are internally defined in the format, every other file
    162.       // however is. The last two bytes of the file are reserved for the version.
    163.       int length = packedLength +Container.getHeaderLength(compression);
    164.       if(chunk.getType()!= EntryType.TYPE_TABLE){
    165.         length +=2;
    166.       }
    167.  
    168.       CachePointer pointer =new CachePointer(chunkId, length);
    169.       pointers.put(MortonCode.mortonCode2i(chunk.getType(), chunk.getEntryId()), pointer);
    170.     }
    171.   }
    172.  
    173.   /**
    174.    * Gets the volumes that are mentioned in all of the chunks.
    175.    *
    176.    * @return the volume ids.
    177.    * @throws IOException an I/O exception was encountered.
    178.    */
    179.   public Set<Integer> getMentionedTypes()throwsIOException{
    180.     Set<Integer> volumes =new HashSet<>();
    181.     EntryChunk chunk =new EntryChunk();
    182.     for(int chunkId =1; chunkId < size(); chunkId++){
    183.       readChunk(chunk, chunkId);
    184.       volumes.add(chunk.getType());
    185.     }
    186.     return volumes;
    187.   }
    188.  
    189.   /**
    190.    * Gets the size of the cache in chunks. The size takes into consideration the reserved EOF
    191.    * chunk so this function will always return a value equal to or greater than one.
    192.    *
    193.    * @return the size of the cache.
    194.    */
    195.   publicint size()throwsIOException{
    196.     return(int)(data.length()/ EntryChunk.LENGTH);
    197.   }
    198. }
    199.  
    200.  
    « Last Edit: April 28, 2016, 04:37:18 AM by _nova »
    Runescape Gambling

     

    Copyright © 2017 MoparScape. All rights reserved.
    Powered by SMFPacks SEO Pro Mod |
    SimplePortal 2.3.5 © 2008-2012, SimplePortal