[WIP] Zeus: Pre-alpha 1 release

Everything about HD, MD, and their mods.

Moderator: Halo Moderators

Locked
Mgalekgolo
Commando
Posts: 2589
Joined: Thu Jun 24, 2010 6:36 am

Re: [WIP] Zeus ~ peer review requested

Post by Mgalekgolo » Wed Nov 16, 2011 10:17 pm

I downloaded it and examined the code. From what I understand, its a good approach. But you should wait for nil to post here <_<. I'm not exactly good at coding yet.
Yarok wrote:There may be a giant dildo protruding from your forehead.
Image

nil
Halo Moderator
Halo Moderator
Posts: 1090
Joined: Sat Jul 05, 2008 8:38 am
Location: null zone

Re: [WIP] Zeus ~ peer review requested

Post by nil » Thu Nov 17, 2011 10:56 am

I'm so far still under the idea that "this project is ridiculous", "it won't be anything", "you're going overboard and should start smaller", "this thread feels like it's trolling me"

Recall, my first post in this thread:
nil wrote:I suppose there isn't harm in trying. Whether or not modzy and dirk are correct about this being too difficult of a goal for him, I'd suggest to Sparky not to create a WIP when having no actual functionality implemented (and if there was functionality already implemented, I'd expect proof). From what I can tell, all he has created so far is an interface without any implementation. There's no harm in trying, but progress follows after implementing things. This is more of a "todo list"; not really reassuring.
Everything I said still stands true. If you looked at his code and didn't interpret it as jargon, you will realize that my old post describes his code exactly.

I know, you want comments on your organization, and I cannot think of any to give. It is extremely insignificant compared to functionality at this point. Software development doesn't work perfectly for "first you plan", "then you implement" - you restructure code and change things all the time, and you waste time if you spend time planning for too long.

Possibly one of the worst feelings in software development is investing too much time on an incomplete project out of your reach and not having anything to show to anyone. One of best feelings is completing a project, which can be surprisingly hard for even simple projects.
I am no longer active to Halo or MGM, and don't guarantee a response on the forums or through email. I will however linger around the discord room for general chatting. It's been fun!

Sparky
Delta Force
Posts: 4194
Joined: Wed Mar 31, 2004 8:59 pm
Location: New Jersey, USA
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Sparky » Thu Nov 17, 2011 11:50 am

Alright, well I'm doing it anyway.

http://pastebin.com/LJP8HJi2 -- Map parsing header file; I welcome comments again so far. I've added more "jargon" in the comments that explain the approach I'm using so far. I do this because it is open source. Comments are appreciated on the one hand and if you still think I can't do it, then calm down and keep watching.

Having analyzed the structure of every tag in the map, I feel like this project is really just a "hop, skip, and a jump" away. Of course, that's not to imply that I know everything about the map files. But what I know is close to enough for what I want to do here, and most of the rest of it -- except maybe the sbsp data that is nigh privy to Modzy, and eventually the shader info that Dirk has been mentioning -- but again, this info is within our community, so I still see this as quite viable. And that previous sentence used grammar improperly, lol... Anyway, back to work.
Either you are groping for answers, or you are asking God and listening to Jesus.

Modzy
Green Beret
Posts: 3058
Joined: Fri Feb 22, 2008 11:06 pm
Location: Portland, OR

Re: [WIP] Zeus ~ peer review requested

Post by Modzy » Thu Nov 17, 2011 4:50 pm

You seem to be confusing the main header data with the index header data, two completely different sections of the file.
The main header holds the following information (taken from OHP):

Code is RB, should be easy to read.

For demo:

Code: Select all

        rs.Position = 2 //0x02
        map_type = rs.ReadUInt8 //0 = SP, 1 = MP, 2 = UI
        rs.Position = &h2C4 //0x2C4
        meta_data_length = rs.ReadUInt32 //Size of tag data
        build_date = rs.Read(&h1F) //in full, the length of these header strings are 0x20, they are 0x1F in demo
        rs.Position = &h588 //0x588
        game_version = rs.ReadUInt32 //The version of Halo; 5 = xbox, 6 = demo, 7 = retail, 609 = CE
        map_name = rs.Read(&h1F) //Map name, max size = 0x1F in Demo; the string is formatted 'funny'
        rs.Position = &h5E8 //0x5E8
        map_length = rs.ReadUInt32 //The size of the map in total
        index_offset = rs.ReadUInt32 //Offset of the index header within the file
For full:

Code: Select all

 //0x0
     game_version = rs.ReadUInt32 //The version of Halo; 5 = xbox, 6 = demo, 7 = retail, 609 = CE
      map_length = rs.ReadUInt32 //The size of the map in total
      garb = rs.Read(4) //Unused
      index_offset = rs.ReadUInt32 //Offset of the index header within the file
      meta_data_length = rs.ReadUInt32 //Size of tag data
      garb = rs.Read(8) //Unused
      map_name = rs.Read(&h20) //Map name, max size = 0x20
      build_date = rs.Read(&h20) //Map build date, max size = 0x20
      map_type = rs.ReadUInt32 //0 = SP, 1 = MP, 2 = UI
The index header is found using the offset in the main header, its structure is such:

Code: Select all

  //Read index header
  rs.Position = index_offset
  memory_offset = rs.ReadUInt32 - &h28 //offset to index array within run-time memory.
  map_magic = memory_offset - index_offset //Universal magic for calculating offsets within the map file which have been modified by version-specific offsets.
  base_tag_id = rs.ReadUInt32 //The ID of the map files' Scenario.
  map_id = rs.ReadUInt32 //Unknown 32-bit integer. Calling it a map ID for now.
  tag_count = rs.ReadUInt32 //The number of items in the index array; tags.
  verticie_count = rs.ReadUInt32 //Number of verticies in model data section.
  verticie_offset = rs.ReadUInt32 //Offset to the start of all verticies in model data section.
  indicie_count = rs.ReadUInt32 //Number of indicies in model data section.
  indicie_offset = rs.ReadUInt32 //Offset to the start of all indicies in model data section, must be modified by verticie offset.
  model_data_length = rs.ReadUInt32 //Size of model data section.
  garb = rs.Read(4) //Unneeded string; "sgat" -- "tags."
Also, I was wondering why you're using "NSNumber" and the like instead of storing your data using a simple long. I find using the fread to parse data and seek offsets is the easiest when you're in a C-type language.

Sparky
Delta Force
Posts: 4194
Joined: Wed Mar 31, 2004 8:59 pm
Location: New Jersey, USA
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Sparky » Thu Nov 17, 2011 6:21 pm

Thanks, and I wasn't confusing the two as it seemed, but thanks for that info, particularly the offsets and type of relevant information.

I'm using NSNumber because I want these things to be more flexible than simple. I might change this later, but that's also my reason for using mutable classes.

Anyway, I've been slightly on the wrong track, and I'll just mention that I'm at the Multi-Document Application tutorial now, so what I'm going to do first is:

- User opens the application, then opens a map file
- Main information of the map file is displayed, somewhat equivalent to Eschaton's main window information about the map file

After this, I'll try to add that preferences window where you specify the location of your map files directories, and that information is stored in a plist file. Although I might not need to do this, who knows... we'll see, I'm not at that point yet. The next step after parsing and displaying the map info -- directly benefitted by your reply here, again thanks -- will be to experiment with parsing a tag (might just start with the contrail tag type, the one I started with while making Sparky's Plugins) and output that information to another window. Basically, the interface is a moot point right now, completely in a rough draft stage, as I experiment with reading and writing tag data from map files.

After I'm done basically translating my Eschaton plugins, I'll probably have to take a hard look at some areas not so familiar to me: sbsp and other models and their collision data (leaves, vertices, etc.)

EDIT: Plans have shifted back to what they were originally with document type definitions. I'm going to define new DTDs for each different type of map file so that a unique file type name and icon can be used with each different version of a halo map file. So in the Finder, Halo CE maps will be distinguishable from Halo Full Mac maps, for example. This should be one of the first benefits of opening and re-saving a map file (without otherwise modifying it) with Zeus. In fact, you might not even need to save the file, but simply open it, and its icon and description should change in the Finder. I'll post the DTDs I use in another topic so that they will be more publicly visible for other app developers to share.
Either you are groping for answers, or you are asking God and listening to Jesus.

nil
Halo Moderator
Halo Moderator
Posts: 1090
Joined: Sat Jul 05, 2008 8:38 am
Location: null zone

Re: [WIP] Zeus ~ peer review requested

Post by nil » Thu Nov 17, 2011 7:01 pm

Modzy wrote:Also, I was wondering why you're using "NSNumber" and the like instead of storing your data using a simple long. I find using the fread to parse data and seek offsets is the easiest when you're in a C-type language.
I thought this once too, but actually, I think it may be better to just load the map in one big gulp in memory. Working in memory is faster than files on disk and it seems less error prone (for example, delete file on disk, your document is still fine since it's loaded in memory). Cocoa, in fact, encourages document-based applications to use NSData.

I'm probably with Modzy with not using NSNumber. You're going to find yourself having to create a new NSNumber each time you modify a NSNumber* (since they're immutable), and you're going to convert to the low level type you want each time you want to do something with it.. Be careful though, long is an bad choice here because sizeof(long) under 64-bit will be 8 bytes and sizeof(long) under 32-bit will be 4 bytes. I'd personally use types like int8_t, int16_t, int32_t, float (as I recall, Halo does not use 64-bit integers or doubles anywhere).
I am no longer active to Halo or MGM, and don't guarantee a response on the forums or through email. I will however linger around the discord room for general chatting. It's been fun!

Modzy
Green Beret
Posts: 3058
Joined: Fri Feb 22, 2008 11:06 pm
Location: Portland, OR

Re: [WIP] Zeus ~ peer review requested

Post by Modzy » Thu Nov 17, 2011 8:26 pm

nil wrote:
Modzy wrote:Also, I was wondering why you're using "NSNumber" and the like instead of storing your data using a simple long. I find using the fread to parse data and seek offsets is the easiest when you're in a C-type language.
I thought this once too, but actually, I think it may be better to just load the map in one big gulp in memory. Working in memory is faster than files on disk and it seems less error prone (for example, delete file on disk, your document is still fine since it's loaded in memory). Cocoa, in fact, encourages document-based applications to use NSData.

I'm probably with Modzy with not using NSNumber. You're going to find yourself having to create a new NSNumber each time you modify a NSNumber* (since they're immutable), and you're going to convert to the low level type you want each time you want to do something with it.. Be careful though, long is an bad choice here because sizeof(long) under 64-bit will be 8 bytes and sizeof(long) under 32-bit will be 4 bytes. I'd personally use types like int8_t, int16_t, int32_t, float (as I recall, Halo does not use 64-bit integers or doubles anywhere).
Loading the map into memory is really only useful if you plan on reading everything in the map. Even then, it's still only useful if you load the map's index and tags into the memory space stated at the beginning of the index header (0x4BF10000 for demo). Then you can define your data types and simply change their pointers to the correct place in memory, instead of actually *reading* from memory and storing the preloaded information twice.

I did not recall the differences of long between 32- and 64-bit architectures (never worked in a 64-bit environment before). Using the int types would be better, as not all Halo data is 32-bit. Bitmasks for instance will be 8-bit and governed by the binary of that 8-bit.

nil
Halo Moderator
Halo Moderator
Posts: 1090
Joined: Sat Jul 05, 2008 8:38 am
Location: null zone

Re: [WIP] Zeus ~ peer review requested

Post by nil » Fri Nov 18, 2011 8:27 am

Efficiency is debatable. Yes, you'll use more memory loading the entire map into a buffer and you may not need all of it and you may have two sets of preloaded information, but so what? On the other hand anyway, writing/reading will be faster.

Either approach, you can define your own data types and you write & read bytes to whatever address you need, whether it's in a file on disk or in some buffer in memory.

Eschaton oddly enough has a feature as I recall that requires the entire map to be loaded into memory (is it map rebuilding?) This feature was added on later of course, but Eschaton may have well just loaded the entire map in memory to begin with rather than have two different ways of loading files, yuck. HHK loads the entire map into memory too. [edit]: Also, If you want to insert data into middle of somewhere, I imagine file I/O is a huge pain (as is done in tag importing?).
Last edited by nil on Fri Nov 18, 2011 9:57 am, edited 2 times in total.
I am no longer active to Halo or MGM, and don't guarantee a response on the forums or through email. I will however linger around the discord room for general chatting. It's been fun!

Dirk Gently
Commando
Posts: 2047
Joined: Sun Oct 21, 2007 2:34 pm
Location: 3C0E9056
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Dirk Gently » Fri Nov 18, 2011 9:01 am

Modzy wrote:
nil wrote:
Modzy wrote:Also, I was wondering why you're using "NSNumber" and the like instead of storing your data using a simple long. I find using the fread to parse data and seek offsets is the easiest when you're in a C-type language.
I thought this once too, but actually, I think it may be better to just load the map in one big gulp in memory. Working in memory is faster than files on disk and it seems less error prone (for example, delete file on disk, your document is still fine since it's loaded in memory). Cocoa, in fact, encourages document-based applications to use NSData.

I'm probably with Modzy with not using NSNumber. You're going to find yourself having to create a new NSNumber each time you modify a NSNumber* (since they're immutable), and you're going to convert to the low level type you want each time you want to do something with it.. Be careful though, long is an bad choice here because sizeof(long) under 64-bit will be 8 bytes and sizeof(long) under 32-bit will be 4 bytes. I'd personally use types like int8_t, int16_t, int32_t, float (as I recall, Halo does not use 64-bit integers or doubles anywhere).
Loading the map into memory is really only useful if you plan on reading everything in the map. Even then, it's still only useful if you load the map's index and tags into the memory space stated at the beginning of the index header (0x4BF10000 for demo). Then you can define your data types and simply change their pointers to the correct place in memory, instead of actually *reading* from memory and storing the preloaded information twice.

I did not recall the differences of long between 32- and 64-bit architectures (never worked in a 64-bit environment before). Using the int types would be better, as not all Halo data is 32-bit. Bitmasks for instance will be 8-bit and governed by the binary of that 8-bit.
You should be using defined types like UInt32 instead of int to make sure you don't run into problems between 64 and 32 bit architecture.

nil
Halo Moderator
Halo Moderator
Posts: 1090
Joined: Sat Jul 05, 2008 8:38 am
Location: null zone

Re: [WIP] Zeus ~ peer review requested

Post by nil » Fri Nov 18, 2011 10:12 am

For info about data size & alignment, see Apple's 64-bit transitioning guide under "Data Type and Alignment": http://developer.apple.com/library/mac/ ... -TPXREF101 - this is OS X specific, and perhaps the compiler and language used has an effect too.

I prefer uint32_t types over UInt32.. Portable and not Apple-specific types..
I am no longer active to Halo or MGM, and don't guarantee a response on the forums or through email. I will however linger around the discord room for general chatting. It's been fun!

Sparky
Delta Force
Posts: 4194
Joined: Wed Mar 31, 2004 8:59 pm
Location: New Jersey, USA
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Sparky » Fri Nov 18, 2011 10:41 am

nil wrote:For info about data size & alignment, see Apple's 64-bit transitioning guide under "Data Type and Alignment": http://developer.apple.com/library/mac/ ... -TPXREF101 - this is OS X specific, and perhaps the compiler and language used has an effect too.

I prefer uint32_t types over UInt32.. Portable and not Apple-specific types..
nil, please fix the link. I don't really like apple docs right now, but perhaps at some point they will be of use to me. In fact, the docs on NSData and NSString are becoming useful to me right now as I try opening and saving map data to disk.
As ugly as this sounds, I want to load the entire map file data into memory and then parse it into an array so that individual array values can be changed dynamically as the modder changes map data in whatever ways the interface allows them to do this, and then finally the entirety of the map data can be saved from the array to disk. Comments?
Either you are groping for answers, or you are asking God and listening to Jesus.

Dirk Gently
Commando
Posts: 2047
Joined: Sun Oct 21, 2007 2:34 pm
Location: 3C0E9056
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Dirk Gently » Fri Nov 18, 2011 4:08 pm

LOL, U DUMB?

Apple docs are some of the best dev docs out there.

nil
Halo Moderator
Halo Moderator
Posts: 1090
Joined: Sat Jul 05, 2008 8:38 am
Location: null zone

Re: [WIP] Zeus ~ peer review requested

Post by nil » Fri Nov 18, 2011 8:47 pm

Sparky wrote:
nil wrote:For info about data size & alignment, see Apple's 64-bit transitioning guide under "Data Type and Alignment": http://developer.apple.com/library/mac/ ... -TPXREF101 - this is OS X specific, and perhaps the compiler and language used has an effect too.

I prefer uint32_t types over UInt32.. Portable and not Apple-specific types..
nil, please fix the link. I don't really like apple docs right now, but perhaps at some point they will be of use to me. In fact, the docs on NSData and NSString are becoming useful to me right now as I try opening and saving map data to disk.
As ugly as this sounds, I want to load the entire map file data into memory and then parse it into an array so that individual array values can be changed dynamically as the modder changes map data in whatever ways the interface allows them to do this, and then finally the entirety of the map data can be saved from the array to disk. Comments?
Yeah, it's not the most link-friendly thing out there. Try this: http://tinyurl.com/6v47bgl (if it doesn't work, just google for it, I mentioned the name and section to look at).

I think the approach seems fine.. I don't know much about halo map structure details though.

@Dirk: Yeah, I never thought about it, but at least for class references I can't think of having used anything better.
I am no longer active to Halo or MGM, and don't guarantee a response on the forums or through email. I will however linger around the discord room for general chatting. It's been fun!

Sparky
Delta Force
Posts: 4194
Joined: Wed Mar 31, 2004 8:59 pm
Location: New Jersey, USA
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Sparky » Fri Nov 18, 2011 10:55 pm

The apple docs are great as a reference only if you already know the language through and through. I'm almost at that point, but it's been quite a struggle getting there. nil was correct about the learning curve being steep. But consider that I learned C++ in high school and read two books on Obj-C and Cocoa already, and also have used the apple docs unsuccessfully (they are not organized to be effective as a tutorial) and early on used the official apple Obj-C programming guide PDF. I already knew what object-oriented programming was from having studied PHP and I also know HTML and other web scripting languages common today like CSS, some JavaScript and XML... well, you know, the whole web programming routine. I never bothered with Ruby because PHP was enough for me; and I never bothered with Obj-C because I was busy with other things, yet during my procedural programming days using C++, I really felt the need for what was Obj-C-in-development. Of course, that was when people were starting to contribute to development of Mac OS X. Before then, I was programming using ResEdit and such... or rather, modding using ResEdit, for the lack of programming resources.

Anyway, you catch my drift. Despite my many years of studying various computer modding tools and programming whatnot, Obj-C has been one of the most difficult languages of which to find quality tutorials. So now I'm finding them online and they are supplementing what I learned from these books I studied during this past year.

One thing that is unique about the way I work is that I can plan and ponder a single subject or topic for a very long time, and the end result is a very fast work pace with confidence of purpose and vision. That's why you've seen so much activity from me these past few days; I'm now at the point of confidence to drive this project home. In comparison, the Eschaton plugins took me about a solid month of work, day in and day out. That was about a year ago, and I garnered confidence of purpose from their necessity at the time, and confidence of vision from through the method of approach (the workflow) I paved for myself. So then it was simply a matter of time. That's quite nearly where I am now with Zeus, and so it's just a matter of time. Days and weeks will go by with me sitting at my computer studying Obj-C and Cocoa tutorials and then immediately applying what I envision from them to Zeus. The speed bumps are caused by the lack of straightforward, easy-to-read tutorials. Of course, I see two such tutorials, but they don't go into much depth in the areas we are concerned. So therefore, I see a need, and you can anticipate that you will see a tutorial of how to make a program like Zeus once it is finished, or at least you will understand how it works through my code comments. Didn't Altimit01 publish some PDFs on how to make a map editor on par with Eschaton? But most of us here have been working in RealBasic.

So my options are twofold: first, spend the rest of my time both learning and coding in Obj-C, or second, make this darn thing in PHP and HTML as a web app where you upload your map file and edit it using a web page hosted on the HDM web server. Of course, the second option would mean indefinite cross-compatibility, but would also mean using some nice bandwidth on the HDM server. But the app would be more obviously integrated with the INCY Online then, and did I mention cross-compatibility? The filesystem is not entirely necessary; but alas, another limitation is the amount of memory the HDM server is allowed to consume in scripts. The more straightforward approach would be Obj-C, so there you have it, and that's what I've been doing.

Still, I appreciate all your comments here. Don't take offense folks, because I want this to be a friendly discourse all the way.
Either you are groping for answers, or you are asking God and listening to Jesus.

Dirk Gently
Commando
Posts: 2047
Joined: Sun Oct 21, 2007 2:34 pm
Location: 3C0E9056
Contact:

Re: [WIP] Zeus ~ peer review requested

Post by Dirk Gently » Fri Nov 18, 2011 11:10 pm

the principal behind Objective-C is really no different from C or C++ for object oriented programming.

purely speculative what is preventing you from making the php version run locally? OS X comes with PHP installed on it already, so why not make it a localized web app?

Locked

Who is online

Users browsing this forum: Ahrefs [Bot], Google [Bot] and 44 guests