Thursday, April 27, 2017

How to download individual files from GitHub enterprise

Hello,

Below instructions how to download individual files from private GitHub repository using powershell.

Assumptions:

  1. Your GitHub repository is hosted at github.mycompany.com
  2. Your organisation name is my-org
  3. Your repository name is my-repo
  4. Path to file you are trying to download is /myfiles/file.txt


Steps


  1. Obtain personal access token for you account by navigating to your account and choosing settings.
  2. Go to Personal Access Token setting and choose "Generate New Token". Copy the resulting token key.
  3. Construct URL to the file you are trying to download in following format: http://github.mycompany.com/api/v3/repos/my-org/my-repo/contents/myfiles/file.txt
  4. Powershell script to download file is below. Values in italic are variables which will be different in your environment

Invoke-WebRequest http://github.mycompany.com/api/v3/repos/my-org/my-repo/contents/myfiles/file.txt -Headers @{"Authorization"="token 8d795936d2c1b2806587719b9b6456bd16549ad8";"Accept"= "application/vnd.github.v3.raw"}

If you need to download entire contents of your master branch then request will look like below

Invoke-WebRequest http://github.mycompany.com/api/v3/repos/my-org/my-repo/zipball/master -Headers @{"Authorization"="token 8d795936d2c1b2806587719b9b6456bd16549ad8"} -OutFile out.zip

Friday, February 24, 2017

IIS WebDav hosting using IIS Manager Users to authentication

Enabling IIS WebDav functionality by using IIS Manager Users


Setting up IIS WebDav functionality is pretty trivial if one to rely on Windows user accounts for authentication but this architecture causing massive issues, namely:

  1. Accounts have to be precreated in Windows and are in fact real Windows accounts with permissions through system. I frequently see people while troubleshooting WebDav authentication issues adding those users to various group (in addition to default Users group) including Administrators account. 
  2. It's difficult to maintain since those users accounts are specific to machine where they live and hence not trivial to extend setup to several servers without keeping all accounts in sync.
Instead we can rely on IIS Manager to store and maintain users which was designed to allow hosting providers to provide remote IIS management functionality to customers. This setup remove all the drawbacks of using Windows users as authentication provider. It's easily scalable (since IIS shared configuration can be used) and do not provide any sort of access to underlying operating system.
Solution consists of 2 DSC scripts below. Instead of using UI to set this up DSC was chosen since it's easily replicated at scale and provide reproducible and consistent behavior.
Prerequisites.ps1 which performs following:

  1. Install basic IIS features
  2. Enabled remote management to enable IIS Manager User features
  3. Install Nuget and chocolatey providers to pull required DSC resources to create website and manipulate NTFS permissions
Startup.ps1 which performs following:

  1. Enables WebDav and neccessary features
  2. Configured IIS Manager to accept both Windows and IIS Manager credentials
  3. Modifies permissions to allow IIS_IUSRS users to read configuration file
  4. Creates website and bindings it to default ports
  5. Create IIS manager users with the password
  6. Modifies IIS configuration to allow WebDav publishing based off IIS Manager credentials provider
  7. Assigns WebDav permissions to newly created users to access website



Prerequisites.ps1
Startup.ps1

Friday, January 20, 2017

Windows Containers Walkthrough

Wiki referenced below provides step by steps instructions how to start working with Windows Containers. It's work in progress but basics steps are already there.


Windows Containers Walktrhough

Friday, February 12, 2016

Running Windows Nano server on QNAP NAS device

How to run Windows Nano server on QNAP



Prerequisites:





Steps.


1.  Download and extract Windows 2016 ISO somewhere on HDD (I use 7-zip for this purposes)
2. Build WIM image by utilizing script below. At the end of the script you shall end up with c:\nanoserver folder with a bunch of subfolders beneath it

$Target_Drive = "C:"  
$cd_drive = "C:\win2016"  
###################  
$NanoTarget = join-path $Target_Drive "Nanoserver"  
$NanoServer = join-path $cd_drive "Nanoserver"  
$Nanosource = join-path $cd_drive "Sources"  
$DismPath = Join-Path $NanoTarget "DISM"  
New-Item -ItemType Directory $NanoTarget  
New-Item -ItemType Directory $DismPath  
foreach ($Filter in "*api*downlevel*.dll","*dism*","*provider*")  
    {  
    Get-ChildItem -Filter $Filter -Path $Nanosource | Copy-Item -Destination $DismPath -PassThru  
    }  
Copy-Item "$NanoServer\*" $NanoTarget -Recurse  

3. Convert VIM image into VHD file with powershell commad below

.\convert-windowsimage.ps1 -SourcePath .\NanoServer.wim -Edition CORESYSTEMSERVER_INSTALL -VHDPath .\nano.vhd -VHDFormat VHD -DiskLayout BIOS

4. You will end up with VHD file in your nano server directory
5. Update you VHD image with OEM drivers below. Make sure "mountdir" folder created first in your build folder.

dism\dism /Mount-Image /ImageFile:.\Nano.vhd /Index:1 /MountDir:.\mountdir
dism\dism /Add-Package /PackagePath:.\packages\Microsoft-NanoServer-OEM-Drivers-Package.cab /Image:.\mountdir  
dism\dism /Add-Package /PackagePath:.\packages\en-US\Microsoft-NanoServer-OEM-Drivers-Package.cab /Image:.\mountdir  
dism\dism /Unmount-Image /MountDir:.\MountDir /Commit  

6. You have fully working VHD now which you can import into Hyper-V if you want, but we need to convert it to qcow2 format used by QNAP by using qemu-img.exe tool

.\qemu\qemu-img.exe  convert -O qcow2 .\nano.vhd dest.img

7. Create new VM in QNAP with this image as HDD and you have yourself a working Nano server running on QNAP


Friday, January 15, 2016

Move any physical/virtual servers to Azure with free tools

Below are steps which can be taken to move physical/virtual servers to Azure. All tools used are freely available.


 Depending which architecture is being moved (physical/virtual) you might start on any of the steps below bypassing some of earlier steps (for example if you want to move Hyper-V managed server). I assume we are moving either from Vmware or physical machine for this flow.

1. Download disk2vhd tool and run on your target machine. Uncheck "VHDx" since Azure supports only VHD files.



2. Create new Virtual Machine and attach generated VHD file to it (Generation 1). Boot machine and uninstall any software which will not be needed in Azure. (Vmware tools for example)



3. Enable firewall for all networks and make exception for remote desktop


4.  If you have system reserved partition then delete it using instructions available on this link


5.  Install or update Hyper-V Integration services components and Azure VM agent (https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-extensions-agent-about/)



6. Make sure you HDD is not bigger then it needs to be once it'll be in Azure. For this to happen you need to defrag disk and move all the files to the start of HDD so you can shrink it to desired size. You will have to do offline defrag in some cases to move all the files to start of HDD. I used Puran Defrag for this purposes.


7. After resize you need to shrink OS partition in Windows to desired final size. 


8. Final 2 steps for VHD is to shrink it and convert it to Fixed size. I used VHDResizer for this purpose.

9. Upload your VHD to Azure storage. I use CloudBerry Explorer. For this you need to register account in Cloudberry by providing account name and key which you can find in Azure portal.
10. Upload your VHD file as Page Blob


11. After upload is complete, go to classic portal and add VHD like below


12. The last step is to create VM based on this VHD


If everything was done right then you will have exact image of you machine running in Azure cloud

Thursday, December 24, 2015

Fixing WinRM startup failure 122 (0x0000007A)

Trying to enable WinRM to use Remoting Session Configuration with RunAsVirtualAccount set to $true.
Session Configuration file is below


Author                        : gs
TranscriptDirectory           : C:\Transcripts\
SessionType                   : Default
SchemaVersion                 : 2.0.0.0
GUID                          : ebf878f7-4829-4f95-8e60-e6999adbbb8a
Architecture                  : 64
Filename                      : %windir%\system32\pwrshplugin.dll
ResourceUri                   : http://schemas.microsoft.com/powershell/jea
MaxConcurrentCommandsPerShell : 1000
Capability                    : {Shell}
xmlns                         : http://schemas.microsoft.com/wbem/wsman/1/config/PluginConfiguration
MaxConcurrentUsers            : 5
lang                          : en-US
SupportsOptions               : true
ProcessIdleTimeoutSec         : 0
ExactMatch                    : true
ConfigFilePath                : C:\Windows\System32\WindowsPowerShell\v1.0\SessionConfig\jea_ebf878f7-4829-4f95-8e60-e6
                                999adbbb8a.pssc
RunAsUser                     :
RunAsVirtualAccountGroups     : Remote Desktop Users;Remote Management Users
IdleTimeoutms                 : 7200000
RunAsVirtualAccount           : True
OutputBufferingMode           : Block
PSVersion                     : 5.0
SecurityDescriptorSddl        : O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;RM)(A;;GA;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxShellsPerUser              : 25
AutoRestart                   : false
MaxShells                     : 25
MaxIdleTimeoutms              : 43200000
Uri                           : http://schemas.microsoft.com/powershell/jea
SDKVersion                    : 2
XmlRenderingType              : text
RunAsPassword                 :
MaxProcessesPerShell          : 15
ParentResourceUri             : http://schemas.microsoft.com/powershell/jea
Enabled                       : True
UseSharedProcess              : false
MaxMemoryPerShellMB           : 1024
Name                          : jea
Permission                    : NT AUTHORITY\INTERACTIVE AccessAllowed, BUILTIN\Administrators AccessAllowed,
                                BUILTIN\Remote Management Users AccessAllowed

After creating endpoint with configuration WinRM failed to start. During process of troubleshooting I enabled Analytic and log which provided 2 non-descriptive error


Trying to decypher this error message produced even less descriptive error explanation
[ADMIN]: PS > winrm helpmsg 122
The data area passed to a system call is too small.
Next tool I tried is Microsoft Message Analyzer and ETW tracing on WinRM provider, which in turn provided almost exactly the same information as Analytic Log (since both sourced from ETW on backend).


After spending hours searching online, eventlog tracing and trying to figure out how to operating Microsoft Message analyzer I turned back to old and trusted friend - procmon.
After capturing File/Registry operations during failure of WinRM service startup and narroring down only to svchost.exe events, I still end up with thousand of entries. I usually go to "count occurance" and check if there were any "Access Denied", "Not Found" results etc but in this case it did not produce any results.
Next stop finding needle in haystack was scrolling all the way down untill you see Windows is trying to deal with failure to start a process(service), that is combing through key relevant to "Error Reporting", as soon as I found those I started working backwards since I assume something happened right before it.
10 lines above it I found what I was looking for.

12:43:45.9389974 PM svchost.exe 12912 RegQueryValue HKLM\SOFTWARE\Policies\Microsoft\Windows\WinRM\Service\DisableRunAs SUCCESS Type: REG_DWORD, Length: 4, Data: 1

Servicehost was checking if WinRM is allowed to use runAs account functionality which had a value of "1" (disabled), changing this value to 0 fixed the issue. Should I have started troubleshooting with procmon I would not have wasted hours on others methods. Lesson learned.


Sunday, December 20, 2015

Fixing issue with Azure file storage and SSL centralized store

Enabling SSL centralized storage via Azure shared File 


 IIS UI will not allow you to input username/password by default provided by Azure for some reason (might be a bug or I'm missing something). Below is workaround which I know for sure works.



 1. Prepend username with bogus domain in UI

 



 2. It will throw an error but will accept username/password and will write to registry. Navigate to HKLM\SOFTWARE\Microsoft\IIS\CentralCertProvider and remove domain name




















SSL centralized store shall work after that.