Sandcastle input flexibility

Jul 25, 2008 at 8:51 PM
I am trying to use Sandcastle to more than just C# code and other languages it said to support. For example, I am programming in a unique language (trust me, it's rare) at work, and was asked to use a documentation generator for it. Basically I will have customized comments (eg time, previous changes, version #, etc) for the class, and would like to use Sandcastle to build the html help files for it. Can this be done? The bottom line is that I need to use Sandcastle for projects ranging from C# to VB to java to some random enterprise language.

Any ideas anyone? Thanks!

(If this has been explained before, please show me the thread to it. Thanks!)
Jul 25, 2008 at 9:48 PM
Edited Jul 25, 2008 at 9:49 PM
Sandcastle basically consist of two tools, the reflected file generator MRefBuilder and the help file assembler or builder BuildAssembler.
You can replace the work of the MRefBuilder by building the reflection file format for your specific language (see the Reflection schema for more information on the format). From there the BuildAssembler will create the help file.

From information made available by a Microsoft developer sometime ago, they do not use the XML Document. So, you can do what they do and use the conceptual help document type codeEntityDocument to produce the same result as the MRefBuilder.
Now, the codeEntityDocument is not well supported by current tools but I am working on it. You should be able to use that to document Java and related languages like C++, Python etc.

Best regards,
Paul.
Jul 25, 2008 at 11:37 PM
Edited Jul 25, 2008 at 11:39 PM
Hi,

It's important to realize though that Sandcastle's presentation styles are designed to document managed code, which means that the reflection files must use a specific schema.  If your custom language doesn't fit into this schema, or if it seems difficult or inappropriate to try to fit your language into the conventions used for managed documentation, then you must create a custom presentation style for your custom reflection schema, at the very least;  e.g., if your language doesn't support the notion of "classes" then you'll find that the reflection schema that Sandcastle uses and the built-in presentation styles might not be appropriate.

Various transformations in the ProductionTransformations folder might not be appropriate for your language either; e.g., ApplyVSDocModel.xsl is used by the vs2005 presentation style to create pseudo topics for things like "All Members", "Overloads", "Methods", "Properties", "Fields" and "Events".  However, programs in the ProductionTools folder, such as XslTransform.exe and BuildAssembler.exe, can still be used as is.

You could start by copying an existing presentation style that you like and then modify the XSL transformations to work with your custom reflection schema.  You'll also probably have to make adjustments, if not a complete overhaul, to the Build Assembler configuration files (sandcastle.config) as they also cater to documentation for managed libraries.  

- Dave
Jul 28, 2008 at 2:56 PM
Thanks guys for the help, I'll look into this and hopefully it'll all work out.
Jul 28, 2008 at 4:31 PM
I want to confirm with you guys if i'm doing this right.

  • So I looked over the Reflection schema, and I was wondering am I suppose to fill up all the elements?
  • Where am I suppose to put the file after building the reflection file format for my language? Are there any samples of reflection files?
  • After I created my schema, do I just run the BuildAssembler.exe on it? Or is there more to it?
  • To davedev: What adjustments do I have to mke to Sandcastle.config?
  • When you say presentation style, did you mean the "hana", "prototype", "shared", "vs2005" presentations? Or did you mean something else?
  • Can I do all the documentation generator in an automated process? ie I can not use the GUI, and create an exe file to do all the processes right?
  • So the bottom line is that 1) i create a reflection xml file 2) apply a xsl transformation to it. Is that what it is, except the process can be done through using the BuildAssembler.exe?

Thanks guys for your help! I am really new to this, but am looking forward to incorporating this technology into my workplace!

Editor
Jul 28, 2008 at 7:40 PM
The schema (C:\Program Files\Sandcastle\Schemas\Reflection\reflection.xsd) will indicate whether or not an element is required or not.  If you see a "minOccurs=" attribute with a value of 1 or more, that usually indicates a required item.  Also, depending on the element, there may be various choices that appear based on some other condition such as the type of item (class, member, field, etc).  You can run MRefBuilder on any exising .NET assembly that you have handy to create example reflection.xml files.  You can also look at the ones in the Sandcastle .\Data folder but be aware that they are the result of running the MRefBuilder generated reflection file through the additional document model XSL transformations to add topic filenames and other stuff like group containers (All Members, Overloads, etc).

The schema isn't really required by the Sandcastle tools so it may just serve as a reference if you create your own tool that produces the reflection information files.  The only non-MRefBuilder generated reflection.xml files that I'm aware of are produced by AjaxDoc.  It produces a reflection.xml file using JavaScript and it follows the existing reflection file schema as I recall.

BuildAssembler doesn't use the schema.  It runs a set of build components defined in sandcastle.config that use the reflection.xml file in combination with other data files to produce the help topics.  The presentation styles transformations (VS2005, Hana, and Prototype) are used to generate the actual topics.  Depending on how you modify the reflection.xml file, you may need to create a custom presentation style to support it.  You may also need to remove build components that don't apply to your language or add new ones that add functionality (i.e. removing unwanted syntax generators or adding new ones).

If you can produce a reflection.xml file and either use an existing presentation style or create a new one, you should be able to use the community tools to automate the process.  For example, the Sandcastle Help File Builder has the ability to use plug-ins that can alter the build process to support non-standard documentation sources such as those produced by AjaxDoc.

Eric
Jul 29, 2008 at 10:34 AM

Hi,

  • So I looked over the Reflection schema, and I was wondering am I suppose to fill up all the elements?

    Eric answered this.  Not all of the elements and attributes may be required but many are important for the built-in presentation styles to work properly.  Go over the various api node types to determine whether the reflection schema is a good fit for your custom language.  If you can at least adapt it, then you may be able to get by using a built-in presentation style with custom content items and a few updates to the XSL transformations, if necessary.

  • Where am I suppose to put the file after building the reflection file format for my language? Are there any samples of reflection files?

    The reflection file is referenced by the Build Assembler configuration file (sandcastle.config) for resolving links.  Search for ResolveReferenceLinksComponent2.  It's also used as the baseline for generating other files using XSL transformations found in the ProductionTransforms folder, such as a build manifest and TOC file.  The manifest is used by Build Assembler and the TOC is used as the baseline for generating compiled help TOCs.  (The TOC may also serve other purposes depending upon the tool that you're using to automate Sandcastle.)

  • After I created my schema, do I just run the BuildAssembler.exe on it? Or is there more to it?

    Eric answered this.

  • To davedev: What adjustments do I have to mke to Sandcastle.config?

    Whatever is necessary to produce HTML help topics as desired.

    Build Assembler runs every component in the sandcastle.config file once, top-down, for each topic in the build manifest.  Typically you'll find components that create in-memory indexes of data, perform a main XSL transformation, resolve shared content items and links, and finally save the topic to disc.  You may be able to get away with just modifying the main XSL transformation but more than likely you'll need to add/remove other components to perform various functions on the reflection data and user comments that you've designed for your custom language.  The ultimate goal of Build Assembler is to execute a build component stack that takes all of the information that has been gathered for a documentation set and produces topic files based on a build manifest.  It is entirely up to you how that transformation is accomplished, including various features that you may want such as shared content and linking, but as I recommended previously, if you start with an existing presentation style then you can make simple adjustments as you need them instead of starting from scratch.  I also recommend reusing the existing build components for things like shared content, indexing, linking and saving, for example.

  • When you say presentation style, did you mean the "hana", "prototype", "shared", "vs2005" presentations? Or did you mean something else?

    That's what I meant :)  Although shared isn't a style - it simply provides objects that are shared by the 3 built-in presentation styles, which you mentioned.

  • Can I do all the documentation generator in an automated process? ie I can not use the GUI, and create an exe file to do all the processes right? 

    You can automate this.  As Eric mentioned, certain tools that automate Sandcastle can be of help, so I'll mention mine :)

    DocProject accepts reflection.xml files and XML documentation files as external sources, so for example you can build help for AjaxDoc output or your custom language by simply adding a reference to these files.  You can also register custom presentation styles in the DocProject config file for seamless integration into Visual Studio and the external GUI.  There are also extensibility features that allow you to plug-in to the DocProject platform for complete control over the project templates, build process and GUIs.

  • So the bottom line is that 1) i create a reflection xml file 2) apply a xsl transformation to it. Is that what it is, except the process can be done through using the BuildAssembler.exe?

    Not quite ;)

    You need a reflection file and probably XML documentation files as well to support user comments.  You also need to apply transformations to the reflection file to produce various other files, including a doc model version, a build manifest and a TOC.  (There are also other requirements if you want to use other features; e.g., versioning.)  Build Assembler can be thought of as the main tool (especially since you won't be needing MRefBuilder for your custom language) but it's probably not the only tool that you'll be using.  It's purpose is to execute a build component stack, which contains build components that transform the information that you've gathered about all sources being documented into individual topics (typically XML-based HTML files).

    - Dave

  • Jul 30, 2008 at 4:59 PM
    Thanks for the replies! Now I have a better understanding on how Sandcastle works!

    Though I have the following questions to ask...

    1) I tried using the Sandcastle Help Builder, and it requires the external source of a dll and user comment xml file. What is the purpose of the dll file? Does it really need a dll file and a comment xml file?

    2) According to the replies, is this what I'm suppose to do to accomplish my objectives?
    • Somehow, create the reflection.xml file, according to the reflection schema
    • Somehow, create the user comments.xml file
    • Apply XSL Transform on the Reflection.xml file (to create doc model version, build manifest and TOC file)
    • Use the Build Assembler to execute the build component stack to ultimately generate a help file.
    So am I correct about what i said above?

    3) Is there any information as to what the reflection schema requirements are for each specifc presentation style? I looked at the schema, saw the "minOccurs=" attributes, but still am clueless as to how or what information i use to populate this schema.

    4) Typically, the comments are to be located right above the methods to describe the specific method. Instead of having the program (ie Visual Studio) generating the user comment xml file for me, is there any other alternative approaches? Or am I limited to Visual Studio?

    5) In my special case, I also have to deal with files that are merely just going to have a bulk of comments at the beginning of my file, that's it. Do you guys have any suggestions as to how I can retrieve this bulk of comments?

    6) In the conceptual help document, I don't understand the whole concept of topics. So am I suppose to use the conceptual format and create a conceptual file to use for Sandcastle or what?


    Thanks for your input guys, really appreciate your help on this topic.
    Jul 30, 2008 at 6:02 PM
    Edited Jul 31, 2008 at 12:49 AM

    >>Adrian 1) What is the purpose of the dll file? Does it really need a dll file and a comment xml file?
    No, the dll is not needed if you are creating the reflection file yourself. SHFB has "comment only" option
    but I do not know if it will work in this case.

    >>Adrian 2) see below...
    >Somehow, create the reflection.xml file, according to the reflection schema
    Yes, if you are doing a completely different language.

    >Somehow, create the user comments.xml file
    No, this is the default comment file in the Sandcastle sample. It is supposed to be the file created by the
    C#,VB.NET or C++/CLI compilers. So, if you are generating the reflection file yourself, you do not need this.

    >Apply XSL Transform on the Reflection.xml file (to create doc model version, build manifest and TOC file)
    Yes, part of the build step.

    >Use the Build Assembler to execute the build component stack to ultimately generate a help file.
    Yes, the main build step, will use the manifest file you created above.

    >>Adrian 3) Is there any information as to what the reflection schema requirements are for each specifc presentation style?
    No, the presentation style do not have any special reflection schema support - the schema is common to all the styles.

    Edited: Deleted - the information is not correct, sorry.

    >>Adrian 6) In the conceptual help document, I don't understand the whole concept of topics.
    Typical developers help will consists of

    • API References - this is where the whole story! of reflection file will do.
    • User Guide - like tutorial, technical papers etc. This is where you will use the conceptual help.

    The only addition here is, the Sandcastle conceptual file format is very extensive with a feature to write the
    API References too, if you do not want to automatically generate it from the source codes.

    One part not mentioned so far is that you will need SyntaxComponent for your language, this will generate the
    functions/methods etc. That is the one you see for each language in a typical Sandcastle output.

    Finally, the third parties tools like SHFB and DocProject are designed to help you compile the help files with
    minimal knowledge of the Sandcastle, so those tools will not be the best way to understand the Sandcastle.
    Try using the command line tools to compile yourself, since your work does require a working knowledge
    of the Sandcastle.
     
    Best regards,
    Paul.

    Editor
    Jul 30, 2008 at 8:36 PM
    In it's standard mode, SHFB requires at least one assembly in order to generate API documentation.  It needs that to have something to pass to MRefBuilder.  However, by creating a plug-in, you can bypass that requirement and supply your own reflection.xml file either directly or by running some other tool.  As long as it ends up in the build's working folder, the remaining build steps will find it.  The AjaxDoc plug-in does this to produce the reflection.xml file via the AjaxDoc web tool.  The "comments only" option that Paul referred to is just for adding additional XML comments files not related to any particular assembly (i.e. one containing namespace comments managed outside of the project).  Running the Sandcastle tools manually is probably a better approach to try out your MRefBuilder replacement rather than coming up with a plug-in first.  The plug-in can come later once you've got the underlying details worked out.

    Paul's comments on points 4 and 5 are misleading.  The XML comments are not related to the production of the reflection.xml file and do not appear in it.  The .NET language compilers produce them off of the comments in the code but you are free to create them in any way you want including maintaining them separate from the code.  As long as the member ID in the XML comments file matches up with an ID in the reflection file, the comments will be included in the topic when it gets generated by BuildAssembler.

    XML comments files only come into play in the BuildAssembler step.  They aren't a requirement but without them, the API topics produced won't contain anything much beyond the syntax and parameter info available in the reflection file.  If modifying the compiler to produce an XML comments file is not possible, you may have to create a tool that parses the source code to pull out the comments.  Whether or not that's part of the tool that creates the reflection file is up to you.

    Eric
    Jul 31, 2008 at 12:51 AM
    >>Eric: Paul's comments on points 4 and 5 are misleading.
    You are right, I have deleted it to avoid a confusion - thank you.

    Best regards,
    Paul.
    Jul 31, 2008 at 2:18 AM
    You'll find links to additional reading material in this article:

    Sandcastle Help
    http://www.codeplex.com/DocProject/Wiki/View.aspx?title=Sandcastle%20Help 

    I'd just like to add that the purpose of the reflection file that MRefBuilder generates (reflection.org) is to provide a structure for the documentation based on the structure of the APIs being documented.  The presentation styles also add their own requirements to the reflection schema, which is where the doc model transformations come in handy to produce the final reflection.xml file.  However, a reflection file for a custom language might not require an additional doc model transformation since it's being designed for only one custom presentation style.

    XML documentation files, on the other hand, provide user documentation that can be taken from code comments or other sources.  In other words, each presentation style's XSL transformations build the structure of each topic based on the reflection information, but the XML documentation files allow you to include user documentation into this auto-generated layout.  For this reason you could still benefit from using documentation files (like comments.xml) for a custom language, regardless of how the reflection file is generated.  You also have the ability to define the schema for your documentation, although I recommend using the C#/VB.NET-compilers' supported schema as a baseline, if it meets your needs, and then support custom elements on top of it.

    One more thing that I'm not sure you're clear on yet: Build Assembler does not produce "compiled" help, in the traditional sense.  It simply produces loose HTML files (topics).  You can take Build Assembler's output (the HTML files) and use them as input to help compilers such as the HTML Help Workshop (.chm) or MS Help 2 (.HxS).  Tools such as DocProject and SHFB automate compilation for you using the topics that Sandcastle produces as input.

    - Dave
    Aug 5, 2008 at 4:12 PM

    So I was discussing with my coworkers, and we would like to know more regarding how we can implement Sandcastle document generator feature with our language. So I want to ask,

    • Does Sandcastle support xml documentation tag only? If so, then I can use xml tags in my custom language too then right?
    • Can I create custom tags for use?
    • Can Sandcastle support java? (I guess this question is irrelevant since if it can support my language, it should be able to support java too.)
    • Does it support the software called Biztalk?
    • Does it support XSLT?
    • If Sandcastle does not support the language, then I cannot use MRefBuilder to parse the file to create a reflection file right?
    • What is reflection.org? is that the reflection file?
    • You guys mentioned API and user guide. So what's the difference between them? Is user guide the presentation style?
    • If i can create a reflection file that is the same as one of the presentation style, then I can just use BuildAssembler to build the API and user guide right? (assuming i have the comment file)
    • Is there any tools out there that you guys recommend to parse the source code to pull out the comments?

    Also, I'm still rather confused with the whole procedure as to how I can use Sandcastle with my language. Correct me if I'm wrong about my steps

    1. Parse my source code to get my comments (save it as a comment.xml file) and create a reflection file (get reflection.org first, then transform it to reflectionl.xml - if i'm using the default presentation style)
    2. Use builder assembler with inputs from reflection.xml file and comment.xml file to create my user guide

    Also, can someone give me a walkthrough on how to create this reflection file? I'm still very lost as to how I can create this reflection file. From looking at the schema, it's really complicated and I'm not sure how I can parse my source code to create this schema.


    Thank you all for helping me out!

    Aug 5, 2008 at 5:52 PM

    Hi, 

    • Does Sandcastle support xml documentation tag only? If so, then I can use xml tags in my custom language too then right?

    The built-in presentation styles use a Build Assembler configuration file and XSL transformations that pull in XML documentation from code comments into the structure being created for each topic.  In your custom configuration and XSL transformations you can do the same, something similar, or something completely proprietary, although just sticking with XML documentation is probably the best choice.

    • Can I create custom tags for use?

    Yes.  The built-in presentation styles use XSL transformations to create topics, so you can add templates for custom tags if you want.  If you're writing your own XSL transformations for your language then you can do whatever you want.

    • Can Sandcastle support java? (I guess this question is irrelevant since if it can support my language, it should be able to support java too.)

    Technically Sandcastle's tools can be used to document any language, although they are designed for documenting managed assemblies.  To document other languages you'll have to do a lot of the work that Sandcastle already does for managed languages.

    • Does it support the software called Biztalk? 

    What kind of support do you mean?

    • Does it support XSLT?

    Sandcastle isn't a single tool - it's several tools and XSLT files that work together.  To ask whether Sandcastle "supports" XSL is kind of like asking whether Sandcastle supports e-mail.  It supports anything that you want.  However, at its core Sandcastle's built-in presentation styles already use XSLT so I guess the answer to your question is yes.

    • If Sandcastle does not support the language, then I cannot use MRefBuilder to parse the file to create a reflection file right?

    Sandcastle doesn't support any languages (except for in the syntax generator).  MRefBuilder creates an XML file that describes the structure of compiled assemblies, not code.

    So if your custom language can produce a managed assembly, then you can use MRefBuilder; otherwise, you can't.

    • What is reflection.org? is that the reflection file?

    Yes, it's a reflection file that is produced by the MRefBuilder executable.  It doesn't have any additional XSL transformations applied yet, such as a doc model, file names or version information.

    • You guys mentioned API and user guide. So what's the difference between them? Is user guide the presentation style?

    There are two types of documentation: Conceptual and Reference.

    Reference documentation (sometimes referred to as API documentation) is automatically generated for managed assemblies and optional XML documentation files.  If you've ever looked at the documentation for something like System.String on MSDN then you were looking at reference documentation.

    Conceptual documentation (sometimes referred to as user documentation) is written using Microsoft Assistance Markup Language (MAML).  It consists of topics like, "How To", "Walk-through", "Troubleshooting", etc.

    • If i can create a reflection file that is the same as one of the presentation style, then I can just use BuildAssembler to build the API and user guide right? (assuming i have the comment file)

    Yes.  XML documentation (the comments file) is optional though.  To build both reference and conceptual documentation you must run Build Assembler twice and then merge the TOCs since the build component stacks (i.e., the Build Assembler configuration files) are different for each type of help: sandcastle.config (reference docs) and conceptual.config (conceptual docs).

    • Is there any tools out there that you guys recommend to parse the source code to pull out the comments?

    If it's source code for a custom language then you'll probably need a custom tool :)

    I'm sure you're aware of this already, but just to be clear: create a managed application that loops through the contents of each file looking for some beginning-of-line token (like /// in C#) or you can use a regular expression.  Then build an in-memory object model that resembles the structure of the target XML documentation file and save the XML to a single file when parsing is complete - I highly recommend using LINQ to XML ;)

    > Also, I'm still rather confused with the whole procedure as to how I can use Sandcastle with my language. Correct me if I'm wrong about my steps [snip]

    It seems like you've got it correct.

    > Also, can someone give me a walk-through on how to create this reflection file?

    Writing a walk-through is not going to be possible without an intimate understanding of your language.  If you have questions about specific things in the schema ask here and we may be able to help; although, Microsoft hasn't provided documentation on the structure or its usage so we may not be able to answer all of your questions.

    - Dave