The documentation for Apache Guacamole doesn't mention anything about configuring a reverse proxy with IIS. You kind of have to translate things from Apache/NGINX to IIS. 

With the default configuration in IIS you will notice that you can get the login page for Apache Guacmole but when you want to start a session it doesn't work and it will give an unrelated error about that there are too many connections. 
The reason for this is that websockets is not installed by default on a IIS server and you have to enable it because Apache Guacamole relies on it to connect to your sessions.

I assume you have enough server knowledge to know where to find the Windows Server Roles & Features section. If you go to the IIS section and expand Application Development you'll find websocket that you can install. 
I also assume you've already installed IIS ARR for the URL Rewrite functionality, otherwise you can download en install it here

Once installed do the following

1. Start IIS
2. On the left side in the Connection View, select your server name
3. Open URL Rewrite
4. On the right side in the Action section, click View Server Variables
5. On the right side in the Action section, click Add
6. Typ in the following value: HTTP_SEC_WEBSOCKET_EXTENSIONS
7. On the right side in the Action section, click Back to rules
8. Go to C:\Windows\System32\inetsrv\config\applicationHost.config 
9. Scroll all the way down to <globalRules> and below </clear /> add the following:
   

   <rule name="ARR_ApacheGuacamole_SSL" enabled="true" patternSyntax="ECMAScript" stopProcessing="true">
  <match url="(.*)" />
  <action type="Rewrite" url="http://Apache Guacamole/{R:0}" />
  <conditions>
    <add input="{HTTPS}" pattern="^ON$" />
    <add input="{HTTP_HOST}" pattern="YOUR SITE HERE" />
   </conditions>
   <serverVariables>
    <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="" />
   </serverVariables>
</rule>

   <rule name="ARR_ApacheGuacamole_HTTPtoHTTPS" enabled="true" stopProcessing="false">
  <match url="(.*)" />
  <conditions>
    <add input="{HTTPS}" pattern="^ON$" />
    <add input="{HTTP_HOST}" pattern="YOUR SITE HERE" />
  </conditions>
  <action type="Rewrite" url="http://Apache Guacamole/{R:0}" />
  <serverVariables>
    <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value=""/>
  </serverVariables>
</rule>

   This creates 2 rules in IIS. One is accept SSL connections and the other is to convert HTTP to HTTPS. This is assuming you have your certs configured on IIS. 
   It is possible to leave it all HTTP, if you want to do that for "testing" reasons, then just add this line:
   <rule name="ARR_ApacheGuacamole" enabled="true" patternSyntax="ECMAScript" stopProcessing="true">
  <match url="(.*)" />
  <action type="Rewrite" url="http://Apache Guacamole/{R:0}" />
  <conditions>
    <add input="{HTTPS}" pattern="^OFF$" />
    <add input="{HTTP_HOST}" pattern="YOUR SITE HERE" />
   </conditions>
   <serverVariables>
    <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="" />
   </serverVariables>
</rule>
10. Save the file
11. Test the connection! Easiest would be with an external connection, but you can also route internal traffic to your proxy. 
    
    
    

Normally you can edit rules in the IIS GUI perfectly fine, but not in this case. This is due to the ServerVariable settings. When you edit this in the GUI you are forced to enter a value, we want this to be empty for it to work. 

You can try this for fun if you want, if you edit it in the GUI and check the file it will show this instead  <set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value=" " enabled=TRUE />
IIS will never pass the traffic onto Guacamole because of this.