This thread will document the final stages of development for the PlayBASIC to DLL conversion tool.  This is a tool to translate PlayBASIC byte code to machine code DLL's.  These DLL's can then be included in your program (bound internally and executed directly from memory) to give you execution speed as fast your system can manage.

    The tool uses the PlayBASIC V1.64P retail compiler (at least) to convert your source code the byte code, the byte code is then translated to assembly and assembled to dll.   This models enables multiple levels of optimizations in the output code.   The PlayBASIC compiler includes many code optimization features from instruction redundancies through to structure caching at byte code level.  The assembly generation stage builds upon this further including things like inlining support with the ability to short cut and further simplify the original byte codes logic, giving a more stream lined and efficient result.

 

---

---

 

---

---

---

Q. Will PlayBASIC To DLL work with the FREE learning editions or demo's

---

       A.  No, it's built as a companion tool for the retail editions.

---

Q. Is this going to be free ?

---

       A.  No.  The product will retail for $49.99 US,  with a cheaper edition available to the PlayBASIC community first (Note: Community Edition released 3rd,June,2014 - Community Special).    You'll of course need a retail version of PlayBASIC for it to be of any use !

---

Q. Are updates to this product FREE ?

---

       A.  Yes, they'll be free for the life of the product.

---

Q. Can I Beta Test or get a Freebie ?

---

       A.  Possibly, but you'd need to really show a high level of PlayBASIC expertise to be considered.

---

Q. Where can find the older PlayBASIC To DLL Blog ?

---

       A.  Like the majority of projects here, this tool started it's life as private project.  Forum members can be read about the early stages of the project   Here (login required)

---

Q. Can I use the DLL's produced with other BASIC's or C ?

---

       A. No ,  the exported code is completely dependent upon PlayBASIC. So without PlayBASIC they won't function.

---

Q. How long does it take to convert PlayBASIC source code to Machine Code DLL?

---

       A. Depends on the size of the program.  But generally it's only few seconds from source code to end machine code dll.   The PlayBASIC compiler does all the really heavy lifting and is capable of compile speed beyond 10,000 lines per second on a single core system.   The end translation to assembly and final assembly passes are routinely inside a second.    

---

Q. What version of PlayBASIC does PB2DLL work with ?

---

       A. It requires a minimum of PlayBASIC V1.64P (or it's betas)

---

Q. Can I use PB2DLL dll's with the Free Learning Editions of PlayBASIC ?

---

       A. No, the learning edition is 3 years behind the current retail edition. If the learning edition is updated to to the V1.64P specification, then you could use DLL's with it..

---

Q. Does PB2DLL optimized my programs code or does it just convert it to machine code ?

---

       A. Yes, PB2DLL does indeed make further optimizations to your program during translation.  The program is optimized twice.  The source code it first translated to byte code using PlayBASIC (retail) compiler, the compiler has many optimizations built in that will reduce the output byte code to the minimum set of operations possible.   When PB2DLL translates your program, it's translating the Byte code representation of your source.  Here it can make further optimizations that the PB compiler can't such as removing the ambiguous nature of some byte code operations or in-lining the operation for you.  Depending upon the operation these short cuts and trim out a massive amount of 'function' calling overhead that would be experienced running the byte code through the the PB runtimes.

  PlayBASIC To Dll  - Support for Globals - Program Re-Composition

         Up until this today the translator focused purely on user functions in the byte code.  So functions named with "DLL_"  would be exported and publicly addressed.   The only problem is that those functions couldn't really set up global data on the dll side.  So the function pretty much had to be standalone.  Which is fine for a lot of stuff, but there are situations where it's easier if the data was handled on the dll side.   So you can just export the code over to  DLL and forget.  

         For example, let's say you want to build a command set like a menu system.  What we could do is pass in/out typed pointers to the menu items as we need.  But this means the user is managing most of the data.   It'd be easier if the DLL side managed the data and just gave us an idiot proof interface to our command set.  So the data is hidden inside the DLL conceptually.  

         Generally for something like this we'd just dimension/create some array that's global to our helper functions, which are then built upon startup with a bunch of  Creation and Get/Set functions to manage the properties of the objects.  

         Eg,

    Type MenuThing
           Name$
           TExt$ 
           X,Y
  etc etc
    EndTYpe

    Dim Menu(0) as MenuThing


Function DLL_NewMenu(GadgetName$,DisplayText$)
      index=getfreecell(_Menu())

      Menu(index) = new MenuThing
      Menu(index).Name$=upper$(GadgetName$)
      Menu(index).Text$  =DisplayText$

EndFunction Index


Function DLL_GetMenuStatus(Index)
     Status=ValidMenu(Index)
endFunction Status


Function DLL_DeleteMenu(Index)
     if ValidMenu(Index)
                Menu(Index) = Null
     endif
endFunction


Psub ValidMenu(Index)
    Status=0
     if Index>0
      if Index<=GetArrayElements(Menu())
                Status=Menu(Index)<>0
      endif
     endif
EndPsub Status

  etc etc

      So what today's build does, is it takes the 'global code'  and builds it into a sub routine that's called when the DLL is initialized by windows.  Basically the same as how C/C++ programs start up.  The only difference is you don't have a forced entry point.   Without this, the 'menu' array in this example would never exist and you'd have to manage it (create it) from inside the sibling functions.  Which isn't that difficult, but this way is really no different that normal PlayBASIC code.   You can of course have your own initialize function as part of your dll.   But the global code in the file will be executed before that.  

  PlayBASIC To Dll  - Select Statement Support

        Unlike most of the control statements Select/Case statement blocks are only now being implemented in the translator.  PlayBASIC's support for  Select Cases is rather unique. Generally select/switch statement blocks are explicitly literal, so cases must be constant.   Which helps the compiler back end when producing the appropriate logic.   The assumption most programmers make, is  their low level compiler is building a jump table, but this actually isn't true.  Visual C/C++ produces a range of different solutions for switch statements, as the input data has to fit the jump table model, even then, it often doesn't .

        In the most BASIC languages (PB included)  select/case block are generally nothing more elaborate IF/EndIF structures. The benefit being the 'select variable' can be cached on register unlike a block of user IF/Then statements.  The caching removes most of memory access when falling through the structure.    Converting this logic directly to assembly is a cake walk and will certainly perform very well.    Hey, if it's good enough for visual C it's good enough for us :)  

        However, the PlayBASIC compiler & runtime times support Variables, expressions, floats  and even strings in case statements.  So you can really mix up the anything you like into them (within reason).   This means the translator has to look ahead at any following case statements when producing code to try and suss out if a register can be cached or not.   The current version doesn't bother with that for a minute, just wanted to get the structure up and running first.

        Here's the current working test code, in this example we're building an Integer select on the A Variable with literal case matching.
       

Function DLL_SelectCase(A,b#,s$)

      
      Select A
         case 0
            print "a=0"
         case 1
            print "a=1"
         case 5 to 2   
            print "a=2 to 5"

         case 10,6,7,8,9
            print "a=6,7,8,9,10"
            
         default
            print "no Match"
            
      EndSelect
   
EndFunction

      The translator already includes generation logic to pre-flips literal case terms and sorts + recast case rows into order.   So in the case 10,6,7,8,9 line, the literals are pre-sorted.  If the set has more than 3 values it inserts a bounds check for you.    So if the select variable is outside the range, it moves on without falling through this block of compares.    


     Edit #1:

        Working on dynamic versions of the select statement block.   The dynamic versions gives the exporter support for variables in case statements.  The  current focus is getting a good range detection without branching.   Have worked out way, but it's behavior is different from the VM.  The VM treats the range as inclusive ->inclusive, but the assembly version is currently inclusive->exclusive.   Which might not sound like a big deal but the behavior then is different between the two.

Function DLL_IntegerSelectCaseTest(A )

         b#=123.456
         Variable=15
         Variable2=20


         r1=50
         r2=60
      
         Select A
            case 0
               result=0
            case 1
               result=1
            case 5 to 2   
               result=2
   
            case 10,6,7,8,9
               result=3
            
            case 66.5
               result=4

            case Variable
               result=5

            case B#
               result=6


            case Variable to 20
               result=7

            case r1 to r2
               result=8

            default
               result=-1
         EndSelect
   
   
EndFunction result

  PlayBASIC To Dll  - Floating Point Select Statement Support

        Today's little chore was to add support for floating point select statements.  Unlike the integer support, floats are pretty chunky in x86 making it difficult to implement boolean styled logic without branches.  Which is pretty annoying given the cost of a miss predicted branch in x86 is _very_ high.    But after an afternoon of testing, it does seem like the only way to get a reasonable result.  Eliminating the branches is possible, but the state handling seems to really impede any benefit, at least that what I get from testing on my test machines.     


        Meaning this is possible in the translator now.

Function DLL_test_Float_select(a#)


   Print "Floating Select VALUE:"+Str$(a#)
   Select a#

       Case 5.7,5.8,5.9
           Print "yeah i'm in th list of options"

       Case 1.5
             Print "yeah i'm the first option"
     
       Case 1.5
             Print " i'm the first option AGAIN ?"

       Case 2.5
           Print "yeah i'm second option"
      
       Case 3.5
           Print "yeah i'm third option"

      Case 7.6 To 66.7
         Print  "with the Range 7.6 To 66.7"

      Case Test1+test2
         Print "Matched the Test Cases"

      Default
         Print "didn't match anything"

 EndSelect
 
 Print ""
EndFunction

       


        After some consideration I'd probably recommend users convert their floating point select case statements to integer.   

         Something like this would probably be faster at runtme. 

Function DLL_test_Float_select(a#)


        Key = Floor(a#*10)

   Print "Floating Select VALUE:"+Str$(a#)
   Select Key

       Case 57,58,59
           Print "yeah i'm in th list of options"

       Case 15
             Print "yeah i'm the first option"
     
       Case 15
             Print " i'm the first option AGAIN ?"

       Case 25
           Print "yeah i'm second option"
      
       Case 35
           Print "yeah i'm third option"

      Case 76 To 667
         Print  "with the Range 7.6 To 66.7"

      Case (Test1+test2)*10
         Print "Matched the Test Cases"

      Default
         Print "didn't match anything"

 EndSelect
 
 Print ""
EndFunction

       

  PlayBASIC To Dll  - String Select Case Statements Supported

        String selects were the only version of the structure missing from the translator, which has now been addressed.  The string exporter isn't too smart and literally equates to a series of string compares, even so, it'll still be faster than the generic VM versions.    What the exporter is really missing is some logic to seek ahead and cache the strings better.   Which would help when a series of the cases can be skipped when a likely match is found, but that'll have to remain on the TODO: list for now.

Function DLL_test_String_Select(a$)

   Print "String Select String: "+a$

   Select lower$(a$)

      Case "hello"
         Print "im option one"

      Case "world"
         Print "im option two"

      Case "a" To "g"
         Print "im from A-G"

      Case "mon","tue","wed","thurs","fri"
         Print "Days of the week I don't like"

      Case "sun","sat"
         Print "The two best days of the week, Sat/Sun"

      Default
      Print "didn't match any options"
   EndSelect 
 Print ""

EndFunction

  PlayBASIC To Dll  - Operators Support Mixes of Data Types


         The AND, OR & XOR operators have been in for a while, but it only supported Integers, where the VM actually supports mixtures of integer and floats.   If one term is a float it's recast to integer for you, but still it supports them.  Thus the translator should behave the same way.

Function DLL_AND(A,B)

   a#=A
   b#=B
   
   print A and 255
   print A# and 255
   print 255 and B
   print 255 and B#
   print A and B
   print A and B#
   print A# and B#
   
EndFunction

 PlayBASIC To Dll  - Flex Rotater

       This is something of a tech demo, much like the fractal render from a while back.   The idea here being to take a bigger chunk of code (in this case an old Amiga demo effect) and and build that into a DLL library.    The routine is an old z rotater styled effect where the spans are rotated and the result is drawn pixel by pixel to the screen.   The render loop is interpolating 800*600 pixels and drawing them via the awfuly generic FastDot.   Moveover the texture fetch is just reading a 2D integer array.   So the code was slapped together out to work, rather than be fast.        

      Running the original routine on the PB VM returns about a 500 plus millisecond refresh on my Athlon system.  The first conversion to DLL cuts that to about 61-62 milliseconds.  I suspect that once the array reading and fastdot are removed, it'll be possible to at least double possible triple that.     The only trouble at the moment is the machine code version isn't acting the same the VM version.  So there's some math operation not behaving the same way.


      Bellow is an example of what the original effect running on the Amiga looks like.

     

  PlayBASIC To Dll  - Flex Rotater Test

           The issues mentioned above turned out to be pretty easy to track down, there was problem with the external trig library using radians rather than degrees and the other main issue was some array accessing of data that didn't exist making it crash.    It's still possible to crash at some arbitrary angles when some operation hits infinity in the  FPU, but that also occurs in the VM.   Beyond that, there's still a couple slight differences in the machine code version, but nothing all that taxing to work around for the time being.

           The rotater is based on linear interpolation, where it drawn strips between pairs of rotated 3D points each each scan line of the display. The spans are longer than the texture width in 3D space, so the UV coords are masked to keep them within the texture space (512 *512), which is what makes it repeat.  Simple to do today, but anything but back in the Amiga days.  

            The main work horse routine is the pixel interpolation, the demo includes 3 versions, there's a version that fetchs the texture from a 2D array and plots the dots via FastDot, as well as version that reads the texture from a bank and plots the dots via fastdot and the 3rd version writes directly to the screen in 32bit pixels.   The last one is quickest in the DLL, but slowest on the VM since it's emulating pixel reading and writing.   So those costs are magnified.    

            The attached code requires PlayBASIC V1.64P beta 18 to work correctly, it'll run in beta 16, but the rotation will stutter.  You might be able to fix that by first converting the angles to radians..


 Attached

hmm that's pretty impressive.
when is PB>Dll coming out then?

  There's still a bit to do, the arrays command sets aren't yet supported for example.  Getting all the side to function the same is pretty time consuming.   Then there's Gui->Doc's->website etc etc.     There's bound to be some version prior to that though.

I would be more than happy to help Beta testing!   ::)

Quote from: monkeybot on December 06, 2013, 06:37:13 AM
I would be more than happy to help Beta testing!   ::)

  It's possible, even probably I might say..  Although,  Why I do have a feeling that my idea of beta testing is little bit different than others. :)

  PlayBASIC To Dll  - Array Library

       The foundation of Arrays and Types have been in for a while now, but the Array command sets are still missing.   There's a few reasons for this, but the main one is that the existing library isn't particular friendly due to how VM dependent it is.    So the only way forward is to build a set of the commands that are free of the VM constraints.    Thankfully I've already got a couple of editions of most of this code in different forms, having tried various constructions over the years.

       One of the major differences between the two sides, is the VM editions decode the VM opcodes as well as performing the actions.  Therefore most instructions have logic in them to solve the passed data based on it's data type.  The machine code editions don't really need such logic, since the generator can solve this during translation.    Cutting a tiny a bit of fat from each process at runtime.    Another way to do that is pre-solve the arrays 'bank' when it's handle is written.

       Arrays in PlayBASIC are considered safe houses.   That's because the runtime always checks the existence of the array data/memory  before it lets you read or write to it.  Even so, it's still possible to crash in certain circumstances, but it's generally much safer for users.   One idea lifted from the VM3 model is to use a structure in the array table that contains not only the data bank of the array, but other information such as it's raw pointer and anything else that may be needed.    This eliminates the need to query the VM's manager about the validity of an array every time.  Much like now, there's probably still some situations where that'd break though.    The upside would be that there's one less function call in every read/write of array/type data.   Which obviously means better performance, without a huge culture shock for programmers.

       So far there's not much to show for an afternoons work with only a couple of query functions working, ran into a crash on exit problem yesterday that put the breaks on my progress.  Turned out to be that the variable table (which has to be rebuilt during export) could sometimes be a bit too small.  So the array functions were writing outside of allocated memory.  Easy to fix, but took a while to diagnose..

Function DLL_Array_Functions(Size)

      Dim Stuff(Size)
      
      if Size>10
         Dim Stuff2(Size)
      endif

      print GetArrayStatus(Stuff())
      print GetArrayStatus(Stuff2())

      print GetArrayElements(Stuff(),1)
      print GetArrayDimensions(Stuff())
      
EndFunction

 

QuoteIt's possible, even probably I might say..  Although,  Why I do have a feeling that my idea of beta testing is little bit different than others.

you will have to create some caveats.

PlayBASIC To Dll  - Array Library Cont.

      Slow going at the moment due to a lot of external pressures that have cropped up besides the normal Xmas mumbo jumbo.  The result is that I can only work on this when free time appears, often very late at night.   So perhaps a couple of hours a day, which isn't much.    None the less, still pressing ahead with the Array library, which basically goes like this.   Write/import an operation and make it friendly for the new array structures, then write translation code for PlayBASIC To DLL,  run some tests and try and correct any wacky results.    It's not really a lot of coding persay,  it's just all the time consuming  junk  around that.   

      The Array library is really built on top of two foundation functions, the DIM and UNDIM commands as you know them.   The DIM does all the heavy lifting to create a nice new fresh array, where as UNDIM does all the heavy lifting to destroy this structure.   Of the two, the UnDim process is the more complicated for the DLL side.   Since when we release an array, it not only releases the main buffer, but any associated buffers with it.   Such as strings in string array for example.     Strings are easy enough but the Types are somewhat problematic as the DLL side needs it's own local cope of the local types.. So another slab of data that needs to be dragged across.  It's only necessary really so the DLL side an build a compatible decoder structure for the central type interface.     


      Here's about where these command are current at.  Pretty basic, but hopefully once the type interface stuff is sorted, progress with this library should be easier.

   Type Dude2
         x,y,z
         s$
   endtype

;   END
Function DLL_Array_Functions(Size)

      Dim Stuff(Size)
      if Size>10
         Dim Stuff2(Size)
      endif
      print GetArrayStatus(Stuff())
      print GetArrayStatus(Stuff2())
      print GetArrayElements(Stuff(),1)
      print GetArrayDimensions(Stuff())


      UNdim Stuff()
      print "Stuff status"
      print GetArrayStatus(Stuff())

      Dim Cool(1000) as Dude2
            
      Dim S$(1000) 
      
      For lp =0 to 1000
               S$(lp)="This String "+Str$(lp)
               Cool(lp)= new dude
               Cool(lp).s$="This String "+Str$(lp)
      next

   ;   undim Cool()
      undim s$()


EndFunction

   

 PlayBASIC To Dll  -  User Define Types.

    Support for UDT's has been in the translator for a few weeks, perhaps even a month now, but the translation is still pretty naive.   Namely the DLL side didn't have any understanding the actual type structures.  So you could create a type and stick them an array on the dll side, but if you tried to delete them or copy them then the dll just didn't have that information available to it.    What it needed is it's own local version of the type descriptions inside the DLL,  which is really a table of how a type structure lays out in memory.   So when that type is clear/deleted, any known buffers such as strings can be released from structure.    Without this, they'd leek strings.

    The last few session have been focused on getting the translator to build the tables required and then filling in the back end code to help initialize everything on startup.    The structures aren't anything all that exciting (as you can see bellow). But they give us all the information necessary to deal with typed structures of memory.
     


       ; ---------------------------------------------------------------------
       UDT_dude2:
       ; ---------------------------------------------------------------------
               dd  String_1
               dd 5
               dd 16
               dd UDT_dude2_Fields


               UDT_dude2_Fields:
                       dd  String_2      ,0      ,0
                       dd  String_3      ,0      ,4
                       dd  String_4      ,4      ,4
                       dd  String_5      ,8      ,4
                       dd  String_6      ,12     ,4




    One of the complexities with this project is that data passing needs to work seamlessly between the VM and any DLL libraries.   So a typed array created on the VM side can be passed to the a DLL function and modified by that function and vice versa.   To do that, you'll need a copy of the structure on both sides of the communication.   So your external dll is built against the structure, and you regular VM code also includes a copy of the same structure.  They have to be identical, if they're not, it'll fail ! -

    For some libraries you won't need to share structures at all, since the library will act like a black box and just give the user a set of high level functions, where all the data is housed internally.    

    I've got the all the components in but it's not running, as currently there's a strange crashing when writing to a newly allocated type, should just a matter of back tracking to find out what's broken.  

      Edit #1

      Finally found the what i think was the cause of type allocation and deletion problems on the DLL,  it just turned out to be a slight difference in the definition structure.   Fine by me as it's working again..  yay.. sleep time