Issue with large arrays.

[LemonWizard]

hi. OKay I've tested this on two machines. One is a dell laptop running windows Vista, and it has about 1 GB of ram.
The other one is a windows XP machine with about a 1.30 Ghz processor I'm not 100% sure though.
And the Win XP pc has almost 5 GB of ram in it. But windows only recognizes the first 3 Gb I believe.

On both machines this code makes playbasic crash. Is it a Syn issue or something else? Is this memory related? I sure hope not!!

PlayBASIC Code: [Select]

#include "input"
dim roomslist$(1000)
dim roomdescriptions$(1000)
dim roomobjects$(1000, 30)
dim roomexpandedswitches(1000)
dim roomobjdescriptions$(1000, 30)
for temp=1 to 1000
   roomslist$(temp)="Room "+str$(temp)
   roomdescriptions$(temp)="Enter your description"
   for temp2=1 to 30
      roomobjects$(temp, temp2)="No item"
      print "Storing data for room number"+str$(temp)+" for object# "+str$(temp2)+" in the array!"
      sync
      cls rgb(0, 0, 0)
      
   next temp2
   roomexpandedswitches(temp)=0 ;false   
next temp   

[kevin]

 Works fine here, I suspect whatever you're seeing is an issue with the code overflowing the PB's memory manager in the learning edition.
 
You can get certainly around this, but without knowing what you're trying to achieve, it's difficult to make a suggestion.

But windows only recognizes the first 3 Gb I believe

.

 There's a number of reason for this.   One is that not all 32/64 bit systems  have 32 address lines.    Which would means not all memory in the system can be addressed.

[LemonWizard]

hey thanks

[LemonWizard]

Okay this issue keeps popping up... with for temp loops that are large.

I have a 1000 for 30 loop... meaning loop 1000 times, and then loop 30 times within it. for each item in the 1000 loop..
I'm trying to read strings in from a text file and this code crashes it...

PlayBASIC Code: [Select]

#include "input"
loaddata:
if folderexist("Save")
            
            if fileexist("Save/Saved.txt")
               readfile("Save/Saved.txt"), 1
               else
                  makefile("Save/Saved.txt"), 10000
                  cls rgb(0,0,0)
                  setcursor 100, 0
                  for temp=1 to 20
                     print "To make sure you get this message, the file had to be created. It didn't previously exist."
               next temp
repeatit:
                  setcursor 300, 0
                  B$=staticinput(">Type yes then press enter to confirm you understand")
                  if B$="yes" then print "okay good you understand.--returning to main loop"
                  if b$ <> "yes" then goto repeatit
                     
            endif
            
         else
            
               makedir "Save"
               
            endif
                  
                  
                  
               
               ;read the data
               
               for temp=1 to 1000
                  roomslist$(temp)=readstring$(1)
                        roomdescriptions$(temp)=readstring$(1)
                        c$=readstring$(1)
                        D=val(C$)
                           roomexpandedswitches(temp)=D
                              ;for temp2=1 to 30
                                 ;roomobjdescriptions$(temp, temp2)=readstring$(1) 
                                 ;   roomobjects$(temp, temp2)=readstring$(1) 
                              ;   next temp2
                           next temp   
                           closefile 1   
                              
                                 

return

keeping in mind I already have the file created...ugh so it just crashes
the code that is commented out is the code I want to work, but it doesn't work... for me.. sigh.

and the code below here is the code that creates that large file

PlayBASIC Code: [Select]

if folderexist("Save")
            
            if fileexist("Save/Saved.txt")
               deletefile("Save/Saved.txt")
               else
                  makefile("Save/Saved.txt"), 10000
            endif
            
         else
            
               makedir "Save"
               
            endif
                  
            
               
               writefile("Save/Saved.txt"), 1
               
               ;save the data
               
               for temp=1 to 1000
                  writestring 1, roomslist$(temp)      
                        writestring 1, roomdescriptions$(temp)
                           writestring 1, str$(roomexpandedswitches(temp) )
                              for temp2=1 to 30
                                 writestring 1, roomobjdescriptions$(temp, temp2)
                                    writestring 1, roomobjects$(temp, temp2)
                                 next temp2
                                 next temp   
                                 
                                 
      
      


   
closefile 1

now what I don't understand is why playbasic keeps crashing :S... if you run the code to create that file first
then try to run the code that loads the data with that temp2 loop...it may or may not crash I'm not sure..

Is this an EOF file error or something else.. like a program stack overflow? I really am lost...


and as for what I'm trying to do
I have 1000 rooms. Their names are stored in roomslist$(index)
I have 1000 room descriptions. Their data is stored in roomdescriptions$(index) and these strings take up entire paragraphs.
I have 1000 rooms, and 30 for each object. So the size of  my array for that is roomobjects$(1000, 30)
The dimensions for my object descriptions array is objectdescriptions$(1000, 30) because there has to be 30 descriptions for each object to..
So...

I want to load all of the data in using readstring(fileindex).
The way it gets written to the file is this order

Roomname
roomdescription
object1name
object1description
object2name
object2description
all the way to 30.
then it restarts with the next room.

Now..
I do this 1000 times. For each room.
...I'll post the saved.txt file so you can see what the data is.




okay..I discovered it isn't the readfile command... for some reason playbasic can't handle assigning something to the array
like this

roomobjects$(temp, temp2)=string$
roomobjectdescriptions$(temp, temp2)=string$

so yeah..that seems to be the issue because I changed those two lines to just
E$=readstring$(1)
F$=readstring$(1)
what's wrong with my code...ugh is it me or is it the cpu I'm running it on.. god

[kevin]

  Creating arrays like this,  dim roomobjects$(1000, 30)  will  create 1000*30 individual (variable length) strings.  That's 30,000 memory allocations the memory manager needs to allocate and manage for you.    In the LE edition of the PB runtime, there's issues where there's max number of individual allocations a program can have.   The easiest way to blow this limit is by creating lots and lots of individual strings like you're doing.  

 However, you don't need to use an array for this at all, a fixed size cache would do the job equally well and only costs one memory allocation.  The cache is one solid block of memory, inside it we store the string data. Strings now have a fixed maximum length though, while they can vary if they're lower the this max, they can't exceed the max.   To read/write strings do this we're using PokeBankString And PeekBankString, with a little math to work out where in the cache each 'string' is stored.  


 The following custom solution simulates the dim roomobjects$(1000, 30) array.  
 

PlayBASIC Code: [Select]

   Constant MaxRooms          =1000
   Constant MaxItemsInRoom      =30
   
   Constant   MaxLengthOfString=256  ; 256 bytes per string in cache


   RoomObjectsArray=AllocRoomObjectsCache()

   For ThisRoom =0 to MaxRooms-1
      For ThisItem=0 to MaxItemsInRoom-1
            ThisString$="String"+STR$(StringCount)
           SetRoomObject(RoomObjectsArray,ThisRoom, ThisItem, ThisString$)
            inc StringCount
      next
   next
   

   Cls 0      
   For ThisRoom =0 to MaxRooms-1
      Row$=""
      For ThisItem=0 to MaxItemsInRoom-1
           Row$=Row$+GetRoomObject(RoomObjectsArray,ThisRoom, ThisItem)+","
      next
      #print Row$
      print Row$

      if GetCursorY()>=550
         Sync
         Waitkey
         Waitnokey
         Cls 0
      endif
   next

      print "DONE"   
      print "Number Of Unique Strings In Array:"+Str$(StringCount)
      print "Size of Array Bank:"+STR$(GetBankSize(RoomObjectsArray))
      
      Sync
      Waitkey
      Waitnokey

Psub AllocRoomObjectsCache()
      MaxNumberOfStringsInCache=(MaxRooms+1)*(MaxitemsInRoom)
      SizeOfCache=MaxNumberOfStringsInCache * MaxLengthOfString
      ThisBank=NewBank(SizeOfCache)   
EndPsub Thisbank


Psub  SetRoomObject(ThisBank,RoomIndex, ItemIndex, ThisString$)
   if RoomIndex>=0 and RoomIndex<MaxRooms
      if ItemIndex>=0 and ItemIndex<MaxItemsInRoom

         Address= RoomIndex * (MaxitemsInRoom*MaxLengthOfString)
         Address= Address+(ItemIndex*MaxLengthOfString)
   
         Size=Len(ThisString$)
         if Size>(MaxLengthOfString-4)
            Size=MaxLengthOfString-4
            ThisString$=Left$(ThisString$,Size)         
         endif
         pokeBankInt ThisBank,Address,Size   
         PokeBankString Thisbank,Address+4,ThisString$,Size
      endif
   endif

EndPsub


Psub  GetRoomObject(ThisBank,RoomIndex, ItemIndex)
   ThisString$=""
   if RoomIndex>=0 and RoomIndex<MaxRooms
      if ItemIndex>=0 and ItemIndex<MaxItemsInRoom

         Address= RoomIndex * (MaxitemsInRoom*MaxLengthOfString)
         Address= Address+(ItemIndex*MaxLengthOfString)
   
         Size=PeekBankInt(ThisBank,Address)
         ThisString$=PeekBankString(Thisbank,Address+4,size)
      
      endif
   endif
EndPsub ThisString$

  Another Approach is to store the rows of individual strings packed together in one string.   So we use a 1D array of 1000 cells in size, each cell is a individual variable length string. So it therefore consumes one memory allocation.   The items can be stored in a delimited fashion,  in this case i'm using the "," character to join all the items that are found in each room into a single string.      This type of approach means that each room could have a varying numbers of objects in them,  plus the object names can be any length.

 

PlayBASIC Code: [Select]

   Constant MaxRooms          =1000
   Constant MaxItemsInRoom      =300

   Dim RoomObjects$(MaxRooms)


   Dim ItemsInThisRoom$(MaxItemsInRoom)


   For ThisRoom =0 to MaxRooms-1
      ; init the items that are going to be places in this room
      For ThisItem=0 to MaxItemsInRoom-1
            ItemsInThisRoom$(ThisItem)="String"+STR$(StringCount)
            inc StringCount
      next
      ;  copy the strings in the ItemsInThisRoom array into the RoomObjects() array
     SetRoomObjects(ThisRoom,  ItemsInThisRoom$())
   next
   

   Cls 0      
   For ThisRoom =0 to MaxRooms-1
      Row$=""
      Count=GetRoomObjects(ThisRoom)

      For ThisItem=0 to Count-1
           Row$=Row$+ItemsInThisRoom$(ThisItem)+","
      next
      #print Row$
      print Row$

      if GetCursorY()>=550
         Sync
         Cls 0
      endif
   next

      print "DONE"   
      print "Number Of Unique Strings In Array:"+Str$(StringCount)
      
      Sync
      Waitkey
      Waitnokey



Psub  SetRoomObjects(RoomIndex,  Theseitems$())
   if RoomIndex>=0 and RoomIndex<MaxRooms
         Row$=""
         For lp=0 to GetArrayElements(TheseItems$(),1)
                  Row$=Row$+TheseItems$(lp)+","
         next
         Row$=TrimRight$(Row$,",")
         RoomObjects$(RoomIndex)=Row$      
   endif
EndPsub


Psub  GetRoomObjects(RoomIndex)
   Count=0
   if RoomIndex>=0 and RoomIndex<MaxRooms
         Count=SplitToArray(RoomObjects$(RoomIndex),",",ItemsInThisRoom$(),0)
   endif
EndPsub Count

 

  In terms of efficiency, the first example is probably the better solution,  given that we can access individual objects within any room without needing to split the room fields apart first.    Although,  that assumes that the program needs constant access to what's in other rooms.   If the program only ever needs to concern itself with the current room the player is in, then the second solution would do fine.

[kevin]

Okay this issue keeps popping up... with for temp loops that are large.

I have a 1000 for 30 loop... meaning loop 1000 times, and then loop 30 times within it. for each item in the 1000 loop..
I'm trying to read strings in from a text file and this code crashes it...


keeping in mind I already have the file created...ugh so it just crashes
the code that is commented out is the code I want to work, but it doesn't work... for me.. sigh.

and the code below here is the code that creates that large file
now what I don't understand is why playbasic keeps crashing :S... if you run the code to create that file first
then try to run the code that loads the data with that temp2 loop...it may or may not crash I'm not sure..

Is this an EOF file error or something else.. like a program stack overflow? I really am lost...

As suggested here  - These are all the same issue,  the number if individual strings is causing the memory allocator to crash in the Learning Edition.  In other words, the PB memory manager has a fixed capacity in the learning edition, therefore if you overflow that, it'll die. 

[kevin]

   This is another way of tackling the same problem. This solution assumes there's a limited number of unique item names in the game world.   So rather than storing an 2D array strings, we us a 2D integer array this time.   So we're not storing the name of the item in the array, but the item names index,  which is a lot more efficient in terms of memory.  

PlayBASIC Code: [Select]

   Constant MaxRooms             =1000
   Constant MaxItemsInRoom         =30
   Constant NumberOfUniqueItems   = 100  ; Number of unique item/object names in game


   // Fill the Item Names array with something to look at.
   Dim ItemName$(NumberOfUniqueItems)
   For ThisItem =0 to NumberOfUniqueItems
      ItemName$(ThisItem)="Item #"+Str$(ThisITem)      
   next


   // Create a 2d array to house the list of object stored within a room
   Dim RoomObjects(MaxRooms,MaxItemsInRoom)


   // run through and fill all the rooms with objects, from the object cache
   For ThisRoom =0 to MaxRooms-1
      ; init the items that are going to be places in this room
      For ThisItem=0 to MaxItemsInRoom-1
            RoomObjects(ThisRoom,ThisItem)=rnd(NumberOfUniqueItems)
            inc StringCount
      next
   next
   
   // Display them to the screen
   Cls 0      
   For ThisRoom =0 to MaxRooms-1
      Row$=""

      For ThisItem=0 to MaxItemsInRoom-1
            ItemIndex=RoomObjects(ThisRoom,ThisItem)   
           Row$=Row$+ItemName$(ItemIndex)+","
      next
      #print Row$
      print Row$

      if GetCursorY()>=550
         Sync
         Cls 0
         waitkey
         waitnokey
      endif
   next

      print "DONE"   
      print "Number Of Unique Strings In Array:"+Str$(StringCount)
      
      Sync
      Waitkey
      Waitnokey