mirror of
https://github.com/actions/python-versions.git
synced 2025-04-03 22:09:38 +00:00
mplement script to build and prepare packages for Python
This commit is contained in:
Maxim Lobanov
10
.gitignore
vendored
Normal file
10
.gitignore
vendored
Normal file
@ -0,0 +1,10 @@
|
||||
# Python Tools for Visual Studio (PTVS)
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# Visual studio / VS Code cache
|
||||
.vs/
|
||||
.vscode/
|
||||
|
||||
# Desktop Service Store
|
||||
.DS_Store
|
||||
76
CODE_OF_CONDUCT.md
Normal file
76
CODE_OF_CONDUCT.md
Normal file
@ -0,0 +1,76 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to make participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, sex characteristics, gender identity and expression,
|
||||
level of experience, education, socio-economic status, nationality, personal
|
||||
appearance, race, religion, or sexual identity and orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies within all project spaces, and it also applies when
|
||||
an individual is representing the project or its community in public spaces.
|
||||
Examples of representing a project or community include using an official
|
||||
project e-mail address, posting via an official social media account, or acting
|
||||
as an appointed representative at an online or offline event. Representation of
|
||||
a project may be further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at opensource@github.com. All
|
||||
complaints will be reviewed and investigated and will result in a response that
|
||||
is deemed necessary and appropriate to the circumstances. The project team is
|
||||
obligated to maintain confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
|
||||
|
||||
[homepage]: https://www.contributor-covenant.org
|
||||
|
||||
For answers to common questions about this code of conduct, see
|
||||
https://www.contributor-covenant.org/faq
|
||||
52
CONTRIBUTING.md
Normal file
52
CONTRIBUTING.md
Normal file
@ -0,0 +1,52 @@
|
||||
## Contributing
|
||||
|
||||
[fork]: https://github.com/actions/python-versions/fork
|
||||
[pr]: https://github.com/actions/python-versions/compare
|
||||
[code-of-conduct]: CODE_OF_CONDUCT.md
|
||||
|
||||
Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
|
||||
|
||||
Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [MIT](LICENSE.md).
|
||||
|
||||
Please note that this project is released with a [Contributor Code of Conduct][code-of-conduct]. By participating in this project you agree to abide by its terms.
|
||||
|
||||
## Submitting a pull request
|
||||
|
||||
1. [Fork][fork] and clone the repository
|
||||
1. Create a new branch: `git checkout -b my-branch-name`
|
||||
1. Make your changes
|
||||
1. Push to your fork and [submit a pull request][pr]
|
||||
1. Make sure that checks in your pull request are green
|
||||
|
||||
Here are a few things you can do that will increase the likelihood of your pull request being accepted:
|
||||
|
||||
- Please include a summary of the change and which issue is fixed. Also include relevant motivation and context.
|
||||
- Follow the style guide for [PowerShell](https://github.com/PoshCode/PowerShellPracticeAndStyle).
|
||||
- Write [good commit messages](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
|
||||
|
||||
## Code structure
|
||||
|
||||
### Directory structure
|
||||
```
|
||||
TOOLCACHE-PYTHON-GENERATION
|
||||
├── azure-pipelines/
|
||||
| └──templates/
|
||||
├── builders/
|
||||
├── helpers/
|
||||
├── installers/
|
||||
└── tests/
|
||||
└──sources/
|
||||
```
|
||||
- `azure-pipelines*` - contains global YAML definitions for build pipelines. Reusable templates for specific jobs are located in `templates` subfolder.
|
||||
- `builders` - contains Python builder classes and functions.
|
||||
- `helpers` - contains global helper functions and functions.
|
||||
- `installers` - contains installation script templates.
|
||||
- `tests` - contains test scripts. Required tests sources are located in `sources` subfolder.
|
||||
|
||||
\* _We use Azure Pipelines because there are a few features that Actions is still missing, we'll move to Actions as soon as possible_.
|
||||
|
||||
## Resources
|
||||
|
||||
- [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
|
||||
- [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
|
||||
- [GitHub Help](https://help.github.com)
|
||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2020 GitHub
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
16
README.md
Normal file
16
README.md
Normal file
@ -0,0 +1,16 @@
|
||||
# Python for Actions
|
||||
This repository contains the code and scripts that we use to build Python packages used in [virtual-environments](https://github.com/actions/virtual-environments) and accessible through the [setup-python](https://github.com/actions/setup-python) Action.
|
||||
File [versions-manifest.json](./versions-manifest.json) contains the list of available and released versions.
|
||||
|
||||
> Caution: this is prepared for and only permitted for use by actions `virtual-environments` and `setup-python` action.
|
||||
|
||||
**Status**: Currently under development and in use for beta and preview actions. This repo is undergoing rapid changes.
|
||||
|
||||
Some versions are pre-installed on [virtual-environments](https://github.com/actions/virtual-environments) images.
|
||||
More versions will (soon!) be available to install on-the-fly through the [`setup-python`](https://github.com/actions/setup-python) action.
|
||||
|
||||
## Adding new versions
|
||||
We are trying to build and release new versions of Python as soon as they are released. Please open an issue if any versions are missing.
|
||||
|
||||
## Contribution
|
||||
Contributions are welcome! See [Contributor's Guide](./CONTRIBUTING.md) for more details about contribution process and code structure
|
||||
3
SECURITY.md
Normal file
3
SECURITY.md
Normal file
@ -0,0 +1,3 @@
|
||||
If you discover a security issue in this repo, please submit it through the [GitHub Security Bug Bounty](https://hackerone.com/github)
|
||||
|
||||
Thanks for helping make GitHub Actions safe for everyone.
|
||||
99
azure-pipelines/build-python-packages.yml
Normal file
99
azure-pipelines/build-python-packages.yml
Normal file
@ -0,0 +1,99 @@
|
||||
name: $(date:yyyyMMdd)$(rev:.r)-Python-$(VERSION)
|
||||
trigger: none
|
||||
pr: none
|
||||
|
||||
stages:
|
||||
- stage: Build_Python_MacOS
|
||||
dependsOn: []
|
||||
variables:
|
||||
VmImage: 'macOS-10.14'
|
||||
Platform: macos-1014
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/build-job.yml
|
||||
|
||||
- stage: Test_Python_MacOS
|
||||
condition: succeeded()
|
||||
dependsOn: Build_Python_MacOS
|
||||
variables:
|
||||
VmImage: 'macOS-10.14'
|
||||
Platform: macos-1014
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/test-job.yml
|
||||
|
||||
- stage: Build_Python_Ubuntu_1604
|
||||
dependsOn: []
|
||||
variables:
|
||||
VmImage: 'ubuntu-16.04'
|
||||
Platform: ubuntu-1604
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/build-job.yml
|
||||
|
||||
- stage: Test_Python_Ubuntu_1604
|
||||
condition: succeeded()
|
||||
dependsOn: Build_Python_Ubuntu_1604
|
||||
variables:
|
||||
VmImage: 'ubuntu-16.04'
|
||||
Platform: ubuntu-1604
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/test-job.yml
|
||||
|
||||
- stage: Build_Python_Ubuntu_1804
|
||||
dependsOn: []
|
||||
variables:
|
||||
VmImage: 'ubuntu-18.04'
|
||||
Platform: ubuntu-1804
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/build-job.yml
|
||||
|
||||
- stage: Test_Python_Ubuntu_1804
|
||||
condition: succeeded()
|
||||
dependsOn: Build_Python_Ubuntu_1804
|
||||
variables:
|
||||
VmImage: 'ubuntu-18.04'
|
||||
Platform: ubuntu-1804
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/test-job.yml
|
||||
|
||||
- stage: Build_Python_X64_Windows
|
||||
dependsOn: []
|
||||
variables:
|
||||
VmImage: 'vs2017-win2016'
|
||||
Platform: windows-2016
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/build-job.yml
|
||||
|
||||
- stage: Test_Python_x64_Windows
|
||||
condition: succeeded()
|
||||
dependsOn: Build_Python_X64_Windows
|
||||
variables:
|
||||
VmImage: 'vs2017-win2016'
|
||||
Platform: windows-2016
|
||||
Architecture: x64
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/test-job.yml
|
||||
|
||||
- stage: Build_Python_x86_Windows
|
||||
dependsOn: []
|
||||
variables:
|
||||
VmImage: 'vs2017-win2016'
|
||||
Platform: windows-2016
|
||||
Architecture: x86
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/build-job.yml
|
||||
|
||||
- stage: Test_Python_x86_Windows
|
||||
condition: succeeded()
|
||||
dependsOn: Build_Python_x86_Windows
|
||||
variables:
|
||||
VmImage: 'vs2017-win2016'
|
||||
Platform: windows-2016
|
||||
Architecture: x86
|
||||
jobs:
|
||||
- template: /azure-pipelines/templates/test-job.yml
|
||||
33
azure-pipelines/run-ci-builds.yml
Normal file
33
azure-pipelines/run-ci-builds.yml
Normal file
@ -0,0 +1,33 @@
|
||||
trigger: none
|
||||
pr:
|
||||
autoCancel: true
|
||||
branches:
|
||||
include:
|
||||
- master
|
||||
paths:
|
||||
exclude:
|
||||
- versions-manifest.json
|
||||
|
||||
jobs:
|
||||
- job: Run_Builds
|
||||
pool:
|
||||
name: Azure Pipelines
|
||||
vmImage: 'ubuntu-latest'
|
||||
|
||||
steps:
|
||||
- checkout: self
|
||||
submodules: true
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Run build'
|
||||
inputs:
|
||||
targetType: filePath
|
||||
filePath: './helpers/azure-devops/run-ci-builds.ps1 '
|
||||
arguments: |
|
||||
-TeamFoundationCollectionUri $(System.TeamFoundationCollectionUri) `
|
||||
-AzureDevOpsProjectName $(System.TeamProject) `
|
||||
-AzureDevOpsAccessToken $(System.AccessToken) `
|
||||
-SourceBranch $(Build.SourceBranch) `
|
||||
-SourceVersion $(Build.SourceVersion) `
|
||||
-ToolVersions "$(PYTHON_VERSIONS)" `
|
||||
-DefinitionId $(DEFINITION_ID)
|
||||
22
azure-pipelines/templates/build-job.yml
Normal file
22
azure-pipelines/templates/build-job.yml
Normal file
@ -0,0 +1,22 @@
|
||||
jobs:
|
||||
- job: Build_Python
|
||||
timeoutInMinutes: 90
|
||||
pool:
|
||||
name: Azure Pipelines
|
||||
vmImage: $(VmImage)
|
||||
steps:
|
||||
- checkout: self
|
||||
submodules: true
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Build Python $(VERSION)'
|
||||
inputs:
|
||||
targetType: filePath
|
||||
filePath: './builders/build-python.ps1'
|
||||
arguments: '-Version $(VERSION) -Platform $(Platform) -Architecture $(Architecture)'
|
||||
|
||||
- task: PublishPipelineArtifact@1
|
||||
displayName: 'Publish Artifact: Python $(VERSION)'
|
||||
inputs:
|
||||
targetPath: '$(Build.ArtifactStagingDirectory)'
|
||||
artifactName: 'python-$(VERSION)-$(Platform)-$(Architecture)'
|
||||
80
azure-pipelines/templates/test-job.yml
Normal file
80
azure-pipelines/templates/test-job.yml
Normal file
@ -0,0 +1,80 @@
|
||||
jobs:
|
||||
- job: Test_Python
|
||||
pool:
|
||||
name: Azure Pipelines
|
||||
vmImage: $(VmImage)
|
||||
steps:
|
||||
- checkout: self
|
||||
submodules: true
|
||||
|
||||
- task: DownloadPipelineArtifact@2
|
||||
inputs:
|
||||
source: 'current'
|
||||
artifact: 'python-$(VERSION)-$(Platform)-$(Architecture)'
|
||||
path: $(Build.BinariesDirectory)
|
||||
|
||||
- task: ExtractFiles@1
|
||||
inputs:
|
||||
archiveFilePatterns: '$(Build.BinariesDirectory)/python-$(VERSION)-$(Platform)-$(Architecture).*'
|
||||
destinationFolder: $(Build.BinariesDirectory)
|
||||
cleanDestinationFolder: false
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Apply build artifact to the local machines'
|
||||
inputs:
|
||||
TargetType: inline
|
||||
script: |
|
||||
if ($env:PLATFORM -match 'windows') { powershell ./setup.ps1 } else { sh ./setup.sh }
|
||||
workingDirectory: '$(Build.BinariesDirectory)'
|
||||
|
||||
- task: UsePythonVersion@0
|
||||
displayName: 'Use Python $(VERSION)'
|
||||
inputs:
|
||||
versionSpec: '$(VERSION)'
|
||||
architecture: '$(Architecture)'
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Verbose sysconfig dump'
|
||||
inputs:
|
||||
TargetType: inline
|
||||
script: |
|
||||
Invoke-Expression "python ./sources/python-config-output.py"
|
||||
workingDirectory: '$(Build.SourcesDirectory)/tests'
|
||||
condition: ne(variables['Platform'], 'windows-2016')
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Verbose python binary links'
|
||||
inputs:
|
||||
TargetType: inline
|
||||
script: |
|
||||
$pythonLocation = which python
|
||||
if ($env:PLATFORM -match 'macos') { otool -L $pythonLocation } else { ldd $pythonLocation }
|
||||
workingDirectory: '$(Build.BinariesDirectory)'
|
||||
condition: ne(variables['Platform'], 'windows-2016')
|
||||
|
||||
- task: PowerShell@2
|
||||
displayName: 'Run tests'
|
||||
inputs:
|
||||
TargetType: inline
|
||||
script: |
|
||||
Install-Module Pester -Force -Scope CurrentUser
|
||||
Import-Module Pester
|
||||
$pesterParams = @{
|
||||
Path="./python-tests.ps1";
|
||||
Parameters=@{
|
||||
Version="$(VERSION)";
|
||||
Platform="$(Platform)";
|
||||
}
|
||||
}
|
||||
Invoke-Pester -Script $pesterParams -OutputFile "test_results.xml" -OutputFormat NUnitXml
|
||||
workingDirectory: '$(Build.SourcesDirectory)/tests'
|
||||
|
||||
- task: PublishTestResults@2
|
||||
displayName: 'Publish test results'
|
||||
inputs:
|
||||
testResultsFiles: '*.xml'
|
||||
testResultsFormat: NUnit
|
||||
searchFolder: 'tests'
|
||||
failTaskOnFailedTests: true
|
||||
testRunTitle: "$(TestRunTitle)"
|
||||
condition: always()
|
||||
75
builders/build-python.ps1
Normal file
75
builders/build-python.ps1
Normal file
@ -0,0 +1,75 @@
|
||||
using module "./builders/win-python-builder.psm1"
|
||||
using module "./builders/ubuntu-python-builder.psm1"
|
||||
using module "./builders/macos-python-builder.psm1"
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Generate Python artifact.
|
||||
|
||||
.DESCRIPTION
|
||||
Main script that creates instance of PythonBuilder and builds of Python using specified parameters.
|
||||
|
||||
.PARAMETER Version
|
||||
Required parameter. The version with which Python will be built.
|
||||
|
||||
.PARAMETER Architecture
|
||||
Optional parameter. The architecture with which Python will be built. Using x64 by default.
|
||||
|
||||
.PARAMETER Platform
|
||||
Required parameter. The platform for which Python will be built.
|
||||
|
||||
#>
|
||||
|
||||
param(
|
||||
[Parameter (Mandatory=$true)][Version] $Version,
|
||||
[Parameter (Mandatory=$true)][string] $Platform,
|
||||
[string] $Architecture = "x64"
|
||||
)
|
||||
|
||||
Import-Module (Join-Path $PSScriptRoot "../helpers" | Join-Path -ChildPath "common-helpers.psm1") -DisableNameChecking
|
||||
Import-Module (Join-Path $PSScriptRoot "../helpers" | Join-Path -ChildPath "nix-helpers.psm1") -DisableNameChecking
|
||||
Import-Module (Join-Path $PSScriptRoot "../helpers" | Join-Path -ChildPath "win-helpers.psm1") -DisableNameChecking
|
||||
|
||||
function Get-PythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Wrapper for class constructor to simplify importing PythonBuilder.
|
||||
|
||||
.DESCRIPTION
|
||||
Create instance of PythonBuilder with specified parameters.
|
||||
|
||||
.PARAMETER Version
|
||||
The version with which Python will be built.
|
||||
|
||||
.PARAMETER Architecture
|
||||
The architecture with which Python will be built.
|
||||
|
||||
.PARAMETER Platform
|
||||
The platform for which Python will be built.
|
||||
|
||||
#>
|
||||
|
||||
param (
|
||||
[version] $Version,
|
||||
[string] $Architecture,
|
||||
[string] $Platform
|
||||
)
|
||||
|
||||
$Platform = $Platform.ToLower()
|
||||
if ($Platform -match 'windows') {
|
||||
$builder = [WinPythonBuilder]::New($Version, $Architecture, $Platform)
|
||||
} elseif ($Platform -match 'ubuntu') {
|
||||
$builder = [UbuntuPythonBuilder]::New($Version, $Architecture, $Platform)
|
||||
} elseif ($Platform -match 'macos') {
|
||||
$builder = [macOSPythonBuilder]::New($Version, $Architecture, $Platform)
|
||||
} else {
|
||||
Write-Host "##vso[task.logissue type=error;] Invalid platform: $Platform"
|
||||
exit 1
|
||||
}
|
||||
|
||||
return $builder
|
||||
}
|
||||
|
||||
### Create Python builder instance, and build artifact
|
||||
$Builder = Get-PythonBuilder -Version $Version -Architecture $Architecture -Platform $Platform
|
||||
$Builder.Build()
|
||||
58
builders/macos-python-builder.psm1
Normal file
58
builders/macos-python-builder.psm1
Normal file
@ -0,0 +1,58 @@
|
||||
using module "./builders/nix-python-builder.psm1"
|
||||
|
||||
class macOSPythonBuilder : NixPythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
MacOS Python builder class.
|
||||
|
||||
.DESCRIPTION
|
||||
Contains methods that required to build macOS Python artifact from sources. Inherited from base NixPythonBuilder.
|
||||
|
||||
.PARAMETER platform
|
||||
The full name of platform for which Python should be built.
|
||||
|
||||
.PARAMETER version
|
||||
The version of Python that should be built.
|
||||
|
||||
#>
|
||||
|
||||
macOSPythonBuilder(
|
||||
[version] $version,
|
||||
[string] $architecture,
|
||||
[string] $platform
|
||||
) : Base($version, $architecture, $platform) { }
|
||||
|
||||
[void] Configure() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Execute configure script with required parameters.
|
||||
#>
|
||||
|
||||
$pythonBinariesLocation = $this.GetFullPythonToolcacheLocation()
|
||||
$configureString = "./configure --prefix=$pythonBinariesLocation --enable-optimizations --enable-shared --with-lto"
|
||||
|
||||
### OS X 10.11, Apple no longer provides header files for the deprecated system version of OpenSSL.
|
||||
### Solution is to install these libraries from a third-party package manager,
|
||||
### and then add the appropriate paths for the header and library files to configure command.
|
||||
### Link to documentation (https://cpython-devguide.readthedocs.io/setup/#build-dependencies)
|
||||
if ($this.Version -lt "3.7.0") {
|
||||
$env:LDFLAGS="-L$(brew --prefix openssl)/lib"
|
||||
$env:CFLAGS="-I$(brew --prefix openssl)/include"
|
||||
} else {
|
||||
$configureString += " --with-openssl=/usr/local/opt/openssl"
|
||||
}
|
||||
|
||||
Execute-Command -Command $configureString
|
||||
}
|
||||
|
||||
[void] PrepareEnvironment() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Prepare system environment by installing dependencies and required packages.
|
||||
#>
|
||||
|
||||
### reinstall header files to Avoid issue with X11 headers on Mojave
|
||||
$pkgName = "/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg"
|
||||
Execute-Command -Command "sudo installer -pkg $pkgName -target /"
|
||||
}
|
||||
}
|
||||
162
builders/nix-python-builder.psm1
Normal file
162
builders/nix-python-builder.psm1
Normal file
@ -0,0 +1,162 @@
|
||||
using module "./builders/python-builder.psm1"
|
||||
|
||||
class NixPythonBuilder : PythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Base Python builder class for *Nix systems.
|
||||
|
||||
.DESCRIPTION
|
||||
Contains methods that required to build Python artifact for *nix systems. Inherited from base PythonBuilder class.
|
||||
|
||||
.PARAMETER version
|
||||
The version of Python that should be built.
|
||||
|
||||
.PARAMETER Platform
|
||||
The type of platform for which Python should be built.
|
||||
|
||||
.PARAMETER PlatformVersion
|
||||
The version of platform for which Python should be built.
|
||||
|
||||
.PARAMETER InstallationTemplateName
|
||||
The name of template that will be used to create installation script for generated Python artifact.
|
||||
|
||||
.PARAMETER InstallationScriptName
|
||||
The name of installation script that will be generated for Python artifact.
|
||||
|
||||
.PARAMETER OutputArtifactName
|
||||
The name of archive with Python binaries that will be generated as part of Python artifact.
|
||||
|
||||
#>
|
||||
|
||||
[string] $InstallationTemplateName
|
||||
[string] $InstallationScriptName
|
||||
[string] $OutputArtifactName
|
||||
|
||||
NixPythonBuilder(
|
||||
[version] $version,
|
||||
[string] $architecture,
|
||||
[string] $platform
|
||||
) : Base($version, $architecture, $platform) {
|
||||
$this.InstallationTemplateName = "nix-setup-template.sh"
|
||||
$this.InstallationScriptName = "setup.sh"
|
||||
|
||||
$this.OutputArtifactName = "python-$Version-$Platform-$Architecture.tar.gz"
|
||||
}
|
||||
|
||||
[uri] GetSourceUri() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Get base Python URI and return complete URI for Python sources.
|
||||
#>
|
||||
|
||||
$base = $this.GetBaseUri()
|
||||
|
||||
return "${base}/$($this.Version)/Python-$($this.Version).tgz"
|
||||
}
|
||||
|
||||
[string] GetPythonBinary() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return name of Python binary.
|
||||
#>
|
||||
|
||||
if ($this.Version.Major -eq 2) { $pythonBinary = "python" } else { $pythonBinary = "python3" }
|
||||
return $pythonBinary
|
||||
}
|
||||
|
||||
[string] Download() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Download Python sources and extract them at temporary work folder. Returns expanded archive location path.
|
||||
#>
|
||||
|
||||
$sourceUri = $this.GetSourceUri()
|
||||
Write-Host "Sources URI: $sourceUri"
|
||||
|
||||
$archiveFilepath = Download-File -Uri $sourceUri -OutputFolder $this.WorkFolderLocation
|
||||
$expandedSourceLocation = Join-Path -Path $this.TempFolderLocation -ChildPath "SourceCode"
|
||||
New-Item -Path $expandedSourceLocation -ItemType Directory
|
||||
|
||||
Extract-TarArchive -ArchivePath $archiveFilepath -OutputDirectory $expandedSourceLocation
|
||||
Write-Debug "Done; Sources location: $expandedSourceLocation"
|
||||
|
||||
return $expandedSourceLocation
|
||||
}
|
||||
|
||||
[void] CreateInstallationScript() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create Python artifact installation script based on template specified in InstallationTemplateName property.
|
||||
#>
|
||||
|
||||
$installationScriptLocation = New-Item -Path $this.WorkFolderLocation -Name $this.InstallationScriptName -ItemType File
|
||||
$installationTemplateLocation = Join-Path -Path $this.InstallationTemplatesLocation -ChildPath $this.InstallationTemplateName
|
||||
|
||||
$installationTemplateContent = Get-Content -Path $installationTemplateLocation -Raw
|
||||
$installationTemplateContent = $installationTemplateContent -f $this.Version.Major, $this.Version.Minor, $this.Version.Build
|
||||
$installationTemplateContent | Out-File -FilePath $installationScriptLocation
|
||||
|
||||
Write-Debug "Done; Installation script location: $installationScriptLocation)"
|
||||
}
|
||||
|
||||
[void] Make() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Executes "make" and "make install" commands for configured build sources. Make output will be writen in build_output.txt located in artifact location folder.
|
||||
#>
|
||||
|
||||
Write-Debug "make Python $($this.Version)-$($this.Architecture) $($this.Platform)"
|
||||
$buildOutputLocation = New-Item -Path $this.WorkFolderLocation -Name "build_output.txt" -ItemType File
|
||||
|
||||
Execute-Command -Command "make 2>&1 | tee $buildOutputLocation" -ErrorAction Continue
|
||||
Execute-Command -Command "make install" -ErrorAction Continue
|
||||
|
||||
Write-Debug "Done; Make log location: $buildOutputLocation"
|
||||
}
|
||||
|
||||
[void] CopyBuildResults() {
|
||||
$buildFolder = $this.GetFullPythonToolcacheLocation()
|
||||
Move-Item -Path "$buildFolder/*" -Destination $this.WorkFolderLocation
|
||||
}
|
||||
|
||||
[void] ArchiveArtifact() {
|
||||
$OutputPath = Join-Path $this.ArtifactFolderLocation $this.OutputArtifactName
|
||||
Create-TarArchive -SourceFolder $this.WorkFolderLocation -ArchivePath $OutputPath
|
||||
}
|
||||
|
||||
[void] Build() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Build Python artifact from sources.
|
||||
#>
|
||||
|
||||
Write-Host "Prepare Python Hostedtoolcache location..."
|
||||
$this.PreparePythonToolcacheLocation()
|
||||
|
||||
Write-Host "Prepare system environment..."
|
||||
$this.PrepareEnvironment()
|
||||
|
||||
Write-Host "Download Python $($this.Version)[$($this.Architecture)] sources..."
|
||||
$sourcesLocation = $this.Download()
|
||||
|
||||
Push-Location -Path $sourcesLocation
|
||||
Write-Host "Configure for $($this.Platform)..."
|
||||
$this.Configure()
|
||||
|
||||
Write-Host "Make for $($this.Platform)..."
|
||||
$this.Make()
|
||||
Pop-Location
|
||||
|
||||
Write-Host "Generate structure dump"
|
||||
New-ToolStructureDump -ToolPath $this.GetFullPythonToolcacheLocation() -OutputFolder $this.WorkFolderLocation
|
||||
|
||||
Write-Host "Copying build results to destination location"
|
||||
$this.CopyBuildResults()
|
||||
|
||||
Write-Host "Create installation script..."
|
||||
$this.CreateInstallationScript()
|
||||
|
||||
Write-Host "Archive artifact..."
|
||||
$this.ArchiveArtifact()
|
||||
}
|
||||
}
|
||||
97
builders/python-builder.psm1
Normal file
97
builders/python-builder.psm1
Normal file
@ -0,0 +1,97 @@
|
||||
class PythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Base Python builder class.
|
||||
|
||||
.DESCRIPTION
|
||||
Base Python builder class that contains general builder methods.
|
||||
|
||||
.PARAMETER Version
|
||||
The version of Python that should be built.
|
||||
|
||||
.PARAMETER Architecture
|
||||
The architecture with which Python should be built.
|
||||
|
||||
.PARAMETER HostedToolcacheLocation
|
||||
The location of hostedtoolcache artifacts. Using system AGENT_TOOLSDIRECTORY variable value.
|
||||
|
||||
.PARAMETER TempFolderLocation
|
||||
The location of temporary files that will be used during Python generation. Using system TEMP directory.
|
||||
|
||||
.PARAMETER WorkFolderLocation
|
||||
The location of generated Python artifact. Using system environment BUILD_STAGINGDIRECTORY variable value.
|
||||
|
||||
.PARAMETER ArtifactFolderLocation
|
||||
The location of generated Python artifact. Using system environment BUILD_BINARIESDIRECTORY variable value.
|
||||
|
||||
.PARAMETER InstallationTemplatesLocation
|
||||
The location of installation script template. Using "installers" folder from current repository.
|
||||
|
||||
#>
|
||||
|
||||
[version] $Version
|
||||
[string] $Architecture
|
||||
[string] $Platform
|
||||
[string] $HostedToolcacheLocation
|
||||
[string] $TempFolderLocation
|
||||
[string] $WorkFolderLocation
|
||||
[string] $ArtifactFolderLocation
|
||||
[string] $InstallationTemplatesLocation
|
||||
|
||||
PythonBuilder ([version] $version, [string] $architecture, [string] $platform) {
|
||||
$this.Version = $version
|
||||
$this.Architecture = $architecture
|
||||
$this.Platform = $platform
|
||||
|
||||
$this.HostedToolcacheLocation = $env:AGENT_TOOLSDIRECTORY
|
||||
$this.TempFolderLocation = $env:BUILD_SOURCESDIRECTORY
|
||||
$this.WorkFolderLocation = $env:BUILD_BINARIESDIRECTORY
|
||||
$this.ArtifactFolderLocation = $env:BUILD_STAGINGDIRECTORY
|
||||
|
||||
$this.InstallationTemplatesLocation = Join-Path -Path $PSScriptRoot -ChildPath "../installers"
|
||||
}
|
||||
|
||||
[uri] GetBaseUri() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return base URI for Python build sources.
|
||||
#>
|
||||
|
||||
return "https://www.python.org/ftp/python"
|
||||
}
|
||||
|
||||
[string] GetPythonToolcacheLocation() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return path to Python hostedtoolcache folder.
|
||||
#>
|
||||
|
||||
return "$($this.HostedToolcacheLocation)/Python"
|
||||
}
|
||||
|
||||
[string] GetFullPythonToolcacheLocation() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return full path to hostedtoolcache Python folder.
|
||||
#>
|
||||
|
||||
$pythonToolcacheLocation = $this.GetPythonToolcacheLocation()
|
||||
return "$pythonToolcacheLocation/$($this.Version)/$($this.Architecture)"
|
||||
}
|
||||
|
||||
[void] PreparePythonToolcacheLocation() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Prepare system hostedtoolcache folder for new Python version.
|
||||
#>
|
||||
$pythonBinariesLocation = $this.GetFullPythonToolcacheLocation()
|
||||
|
||||
if (Test-Path $pythonBinariesLocation) {
|
||||
Write-Host "Purge $pythonBinariesLocation folder..."
|
||||
Remove-Item $pythonBinariesLocation -Recurse -Force
|
||||
} else {
|
||||
Write-Host "Create $pythonBinariesLocation folder..."
|
||||
New-Item -ItemType Directory -Path $pythonBinariesLocation
|
||||
}
|
||||
}
|
||||
}
|
||||
85
builders/ubuntu-python-builder.psm1
Normal file
85
builders/ubuntu-python-builder.psm1
Normal file
@ -0,0 +1,85 @@
|
||||
using module "./builders/nix-python-builder.psm1"
|
||||
|
||||
class UbuntuPythonBuilder : NixPythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Ubuntu Python builder class.
|
||||
|
||||
.DESCRIPTION
|
||||
Contains methods that required to build Ubuntu Python artifact from sources. Inherited from base NixPythonBuilder.
|
||||
|
||||
.PARAMETER platform
|
||||
The full name of platform for which Python should be built.
|
||||
|
||||
.PARAMETER version
|
||||
The version of Python that should be built.
|
||||
|
||||
#>
|
||||
|
||||
UbuntuPythonBuilder(
|
||||
[version] $version,
|
||||
[string] $architecture,
|
||||
[string] $platform
|
||||
) : Base($version, $architecture, $platform) { }
|
||||
|
||||
[void] Configure() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Execute configure script with required parameters.
|
||||
#>
|
||||
|
||||
$pythonBinariesLocation = $this.GetFullPythonToolcacheLocation()
|
||||
|
||||
### To build Python with SO we must pass full path to lib folder to the linker
|
||||
$env:LDFLAGS="-Wl,--rpath=${pythonBinariesLocation}/lib"
|
||||
$configureString = "./configure --prefix=$pythonBinariesLocation --enable-shared --enable-optimizations"
|
||||
|
||||
if ($this.Version -lt "3.0.0") {
|
||||
### Compile with ucs4 for Python 2.x. On 3.x, ucs4 is enabled by default
|
||||
$configureString += " --enable-unicode=ucs4"
|
||||
}
|
||||
|
||||
Execute-Command -Command $configureString
|
||||
}
|
||||
|
||||
[void] PrepareEnvironment() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Prepare system environment by installing dependencies and required packages.
|
||||
#>
|
||||
|
||||
if (($this.Version -gt "3.0.0") -and ($this.Version -lt "3.5.3")) {
|
||||
Write-Host "Python3 versions lower than 3.5.3 are not supported"
|
||||
exit 1
|
||||
}
|
||||
|
||||
### Compile with tkinter support
|
||||
if ($this.Version -gt "3.0.0") {
|
||||
$tkinterInstallString = "sudo apt-get install -y --allow-downgrades python3-tk tk-dev"
|
||||
} else {
|
||||
$tkinterInstallString = "sudo apt install -y python-tk tk-dev"
|
||||
}
|
||||
Execute-Command -Command $tkinterInstallString
|
||||
|
||||
### Install dependent packages
|
||||
@(
|
||||
"make",
|
||||
"build-essential",
|
||||
"libssl-dev",
|
||||
"zlib1g-dev",
|
||||
"libbz2-dev",
|
||||
"libsqlite3-dev",
|
||||
"libncursesw5-dev",
|
||||
"libreadline-dev",
|
||||
"libgdbm-dev"
|
||||
) | ForEach-Object {
|
||||
Execute-Command -Command "sudo apt install -y $_"
|
||||
}
|
||||
|
||||
if ($this.Platform -ne "ubuntu-1604") {
|
||||
### On Ubuntu-1804, libgdbm-compat-dev has older modules that are no longer in libgdbm-dev
|
||||
Execute-Command -Command "sudo apt install -y libgdbm-compat-dev"
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
141
builders/win-python-builder.psm1
Normal file
141
builders/win-python-builder.psm1
Normal file
@ -0,0 +1,141 @@
|
||||
using module "./builders/python-builder.psm1"
|
||||
|
||||
class WinPythonBuilder : PythonBuilder {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Base Python builder class for Windows systems.
|
||||
|
||||
.DESCRIPTION
|
||||
Contains methods required for build Windows Python artifact. Inherited from base PythonBuilder class.
|
||||
|
||||
.PARAMETER version
|
||||
The version of Python that should be built.
|
||||
|
||||
.PARAMETER architecture
|
||||
The architecture with which Python should be built.
|
||||
|
||||
.PARAMETER InstallationTemplateName
|
||||
The name of installation script template that will be used in generated artifact.
|
||||
|
||||
.PARAMETER InstallationScriptName
|
||||
The name of generated installation script.
|
||||
|
||||
#>
|
||||
|
||||
[string] $InstallationTemplateName
|
||||
[string] $InstallationScriptName
|
||||
[string] $OutputArtifactName
|
||||
|
||||
WinPythonBuilder(
|
||||
[version] $version,
|
||||
[string] $architecture,
|
||||
[string] $platform
|
||||
) : Base($version, $architecture, $platform) {
|
||||
$this.InstallationTemplateName = "win-setup-template.ps1"
|
||||
$this.InstallationScriptName = "setup.ps1"
|
||||
$this.OutputArtifactName = "python-$Version-$Platform-$Architecture.zip"
|
||||
}
|
||||
|
||||
[string] GetPythonExtension() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return extension for required version of Python executable.
|
||||
#>
|
||||
|
||||
$extension = if ($this.Version -lt "3.5" -and $this.Version -ge "2.5") { ".msi" } else { ".exe" }
|
||||
|
||||
return $extension
|
||||
}
|
||||
|
||||
[string] GetArchitectureExtension() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Return architecture suffix for Python executable.
|
||||
#>
|
||||
|
||||
$ArchitectureExtension = ""
|
||||
if ($this.Architecture -eq "x64") {
|
||||
if ($this.Version -ge "3.5") {
|
||||
$ArchitectureExtension = "-amd64"
|
||||
} else {
|
||||
$ArchitectureExtension = ".amd64"
|
||||
}
|
||||
}
|
||||
|
||||
return $ArchitectureExtension
|
||||
}
|
||||
|
||||
[uri] GetSourceUri() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Get base Python URI and return complete URI for Python installation executable.
|
||||
#>
|
||||
|
||||
$base = $this.GetBaseUri()
|
||||
$architecture = $this.GetArchitectureExtension()
|
||||
$extension = $this.GetPythonExtension()
|
||||
|
||||
$uri = "${base}/$($this.Version)/python-$($this.Version)${architecture}${extension}"
|
||||
|
||||
return $uri
|
||||
}
|
||||
|
||||
[string] Download() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Download Python installation executable into artifact location.
|
||||
#>
|
||||
|
||||
$sourceUri = $this.GetSourceUri()
|
||||
|
||||
Write-Host "Sources URI: $sourceUri"
|
||||
$sourcesLocation = Download-File -Uri $sourceUri -OutputFolder $this.WorkFolderLocation
|
||||
Write-Debug "Done; Sources location: $sourcesLocation"
|
||||
|
||||
return $sourcesLocation
|
||||
}
|
||||
|
||||
[void] CreateInstallationScript() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create Python artifact installation script based on specified template.
|
||||
#>
|
||||
|
||||
$sourceUri = $this.GetSourceUri()
|
||||
$pythonExecName = [IO.path]::GetFileName($sourceUri.AbsoluteUri)
|
||||
$installationTemplateLocation = Join-Path -Path $this.InstallationTemplatesLocation -ChildPath $this.InstallationTemplateName
|
||||
$installationTemplateContent = Get-Content -Path $installationTemplateLocation -Raw
|
||||
$installationScriptLocation = New-Item -Path $this.WorkFolderLocation -Name $this.InstallationScriptName -ItemType File
|
||||
|
||||
$variablesToReplace = @{
|
||||
"{{__ARCHITECTURE__}}" = $this.Architecture;
|
||||
"{{__VERSION__}}" = $this.Version;
|
||||
"{{__PYTHON_EXEC_NAME__}}" = $pythonExecName
|
||||
}
|
||||
|
||||
$variablesToReplace.keys | ForEach-Object { $installationTemplateContent = $installationTemplateContent.Replace($_, $variablesToReplace[$_]) }
|
||||
$installationTemplateContent | Out-File -FilePath $installationScriptLocation
|
||||
Write-Debug "Done; Installation script location: $installationScriptLocation)"
|
||||
}
|
||||
|
||||
[void] ArchiveArtifact() {
|
||||
$OutputPath = Join-Path $this.ArtifactFolderLocation $this.OutputArtifactName
|
||||
Create-SevenZipArchive -SourceFolder $this.WorkFolderLocation -ArchivePath $OutputPath
|
||||
}
|
||||
|
||||
[void] Build() {
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Generates Python artifact from downloaded Python installation executable.
|
||||
#>
|
||||
|
||||
Write-Host "Download Python $($this.Version) [$($this.Architecture)] executable..."
|
||||
$this.Download()
|
||||
|
||||
Write-Host "Create installation script..."
|
||||
$this.CreateInstallationScript()
|
||||
|
||||
Write-Host "Archive artifact"
|
||||
$this.ArchiveArtifact()
|
||||
}
|
||||
}
|
||||
89
helpers/azure-devops/azure-devops-api.ps1
Normal file
89
helpers/azure-devops/azure-devops-api.ps1
Normal file
@ -0,0 +1,89 @@
|
||||
class AzureDevOpsApi
|
||||
{
|
||||
[string] $BaseUrl
|
||||
[string] $RepoOwner
|
||||
[object] $AuthHeader
|
||||
|
||||
AzureDevOpsApi(
|
||||
[string] $TeamFoundationCollectionUri,
|
||||
[string] $ProjectName,
|
||||
[string] $AccessToken
|
||||
) {
|
||||
$this.BaseUrl = $this.BuildBaseUrl($TeamFoundationCollectionUri, $ProjectName)
|
||||
$this.AuthHeader = $this.BuildAuth($AccessToken)
|
||||
}
|
||||
|
||||
[object] hidden BuildAuth([string]$AccessToken) {
|
||||
if ([string]::IsNullOrEmpty($AccessToken)) {
|
||||
return $null
|
||||
}
|
||||
return @{
|
||||
Authorization = "Bearer $AccessToken"
|
||||
}
|
||||
}
|
||||
|
||||
[string] hidden BuildBaseUrl([string]$TeamFoundationCollectionUri, [string]$ProjectName) {
|
||||
return "${TeamFoundationCollectionUri}/${ProjectName}/_apis"
|
||||
}
|
||||
|
||||
[object] QueueBuild([string]$ToolVersion, [string]$SourceBranch, [string]$SourceVersion, [UInt32]$DefinitionId){
|
||||
$url = "build/builds"
|
||||
|
||||
# The content of parameters field should be a json string
|
||||
$buildParameters = @{ VERSION = $ToolVersion } | ConvertTo-Json
|
||||
|
||||
$body = @{
|
||||
definition = @{
|
||||
id = $DefinitionId
|
||||
}
|
||||
sourceBranch = $SourceBranch
|
||||
sourceVersion = $SourceVersion
|
||||
parameters = $buildParameters
|
||||
} | ConvertTo-Json
|
||||
|
||||
return $this.InvokeRestMethod($url, 'POST', $body)
|
||||
}
|
||||
|
||||
[object] GetBuildInfo([UInt32]$BuildId){
|
||||
$url = "build/builds/$BuildId"
|
||||
|
||||
return $this.InvokeRestMethod($url, 'GET', $null)
|
||||
}
|
||||
|
||||
[string] hidden BuildUrl([string]$Url) {
|
||||
return "$($this.BaseUrl)/${Url}/?api-version=5.1"
|
||||
}
|
||||
|
||||
[object] hidden InvokeRestMethod(
|
||||
[string] $Url,
|
||||
[string] $Method,
|
||||
[string] $Body
|
||||
) {
|
||||
$requestUrl = $this.BuildUrl($Url)
|
||||
$params = @{
|
||||
Method = $Method
|
||||
ContentType = "application/json"
|
||||
Uri = $requestUrl
|
||||
Headers = @{}
|
||||
}
|
||||
if ($this.AuthHeader) {
|
||||
$params.Headers += $this.AuthHeader
|
||||
}
|
||||
if (![string]::IsNullOrEmpty($body)) {
|
||||
$params.Body = $Body
|
||||
}
|
||||
|
||||
return Invoke-RestMethod @params
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
function Get-AzureDevOpsApi {
|
||||
param (
|
||||
[string] $TeamFoundationCollectionUri,
|
||||
[string] $ProjectName,
|
||||
[string] $AccessToken
|
||||
)
|
||||
|
||||
return [AzureDevOpsApi]::New($TeamFoundationCollectionUri, $ProjectName, $AccessToken)
|
||||
}
|
||||
44
helpers/azure-devops/build-info.ps1
Normal file
44
helpers/azure-devops/build-info.ps1
Normal file
@ -0,0 +1,44 @@
|
||||
Import-Module (Join-Path $PSScriptRoot "azure-devops-api.ps1")
|
||||
|
||||
class BuildInfo
|
||||
{
|
||||
[AzureDevOpsApi] $AzureDevOpsApi
|
||||
[String] $Name
|
||||
[UInt32] $Id
|
||||
[String] $Status
|
||||
[String] $Result
|
||||
[String] $Link
|
||||
|
||||
BuildInfo([AzureDevOpsApi] $AzureDevOpsApi, [object] $Build)
|
||||
{
|
||||
$this.AzureDevOpsApi = $AzureDevOpsApi
|
||||
$this.Id = $Build.id
|
||||
$this.Name = $Build.buildNumber
|
||||
$this.Link = $Build._links.web.href
|
||||
$this.Status = $Build.status
|
||||
$this.Result = $Build.result
|
||||
}
|
||||
|
||||
[boolean] IsFinished() {
|
||||
return ($this.Status -eq "completed") -or ($this.Status -eq "cancelling")
|
||||
}
|
||||
|
||||
[boolean] IsSuccess() {
|
||||
return $this.Result -eq "succeeded"
|
||||
}
|
||||
|
||||
[void] UpdateBuildInfo() {
|
||||
$buildInfo = $this.AzureDevOpsApi.GetBuildInfo($this.Id)
|
||||
$this.Status = $buildInfo.status
|
||||
$this.Result = $buildInfo.result
|
||||
}
|
||||
}
|
||||
|
||||
function Get-BuildInfo {
|
||||
param (
|
||||
[AzureDevOpsApi] $AzureDevOpsApi,
|
||||
[object] $Build
|
||||
)
|
||||
|
||||
return [BuildInfo]::New($AzureDevOpsApi, $Build)
|
||||
}
|
||||
94
helpers/azure-devops/run-ci-builds.ps1
Normal file
94
helpers/azure-devops/run-ci-builds.ps1
Normal file
@ -0,0 +1,94 @@
|
||||
param (
|
||||
[Parameter(Mandatory)] [string] $TeamFoundationCollectionUri,
|
||||
[Parameter(Mandatory)] [string] $AzureDevOpsProjectName,
|
||||
[Parameter(Mandatory)] [string] $AzureDevOpsAccessToken,
|
||||
[Parameter(Mandatory)] [string] $SourceBranch,
|
||||
[Parameter(Mandatory)] [string] $ToolVersions,
|
||||
[Parameter(Mandatory)] [UInt32] $DefinitionId,
|
||||
[string] $SourceVersion
|
||||
)
|
||||
|
||||
Import-Module (Join-Path $PSScriptRoot "azure-devops-api.ps1")
|
||||
Import-Module (Join-Path $PSScriptRoot "build-info.ps1")
|
||||
|
||||
function Queue-Builds {
|
||||
param (
|
||||
[Parameter(Mandatory)] [AzureDevOpsApi] $AzureDevOpsApi,
|
||||
[Parameter(Mandatory)] [string] $ToolVersions,
|
||||
[Parameter(Mandatory)] [string] $SourceBranch,
|
||||
[Parameter(Mandatory)] [string] $SourceVersion,
|
||||
[Parameter(Mandatory)] [string] $DefinitionId
|
||||
)
|
||||
|
||||
[BuildInfo[]]$queuedBuilds = @()
|
||||
|
||||
$ToolVersions.Split(',') | ForEach-Object {
|
||||
$version = $_.Trim()
|
||||
Write-Host "Queue build for $version..."
|
||||
$queuedBuild = $AzureDevOpsApi.QueueBuild($version, $SourceBranch, $SourceVersion, $DefinitionId)
|
||||
$buildInfo = Get-BuildInfo -AzureDevOpsApi $AzureDevOpsApi -Build $queuedBuild
|
||||
Write-Host "Queued build: $($buildInfo.Link)"
|
||||
$queuedBuilds += $buildInfo
|
||||
}
|
||||
|
||||
return $queuedBuilds
|
||||
}
|
||||
|
||||
function Wait-Builds {
|
||||
param (
|
||||
[Parameter(Mandatory)] [BuildInfo[]] $Builds
|
||||
)
|
||||
|
||||
$timeoutBetweenRefreshSec = 30
|
||||
|
||||
do {
|
||||
# If build is still running - refresh its status
|
||||
foreach($build in $builds) {
|
||||
if (!$build.IsFinished()) {
|
||||
$build.UpdateBuildInfo()
|
||||
|
||||
if ($build.IsFinished()) {
|
||||
Write-Host "The $($build.Name) build was completed: $($build.Link)"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$runningBuildsCount = ($builds | Where-Object { !$_.IsFinished() }).Length
|
||||
|
||||
Start-Sleep -Seconds $timeoutBetweenRefreshSec
|
||||
} while($runningBuildsCount -gt 0)
|
||||
}
|
||||
|
||||
function Make-BuildsOutput {
|
||||
param (
|
||||
[Parameter(Mandatory)] [BuildInfo[]] $Builds
|
||||
)
|
||||
|
||||
Write-Host "Builds info:"
|
||||
$builds | Format-Table -AutoSize -Property Name,Id,Status,Result,Link | Out-String -Width 10000
|
||||
|
||||
# Return exit code based on status of builds
|
||||
$failedBuilds = ($builds | Where-Object { !$_.IsSuccess() })
|
||||
if ($failedBuilds.Length -ne 0) {
|
||||
Write-Host "##vso[task.logissue type=error;]Builds failed"
|
||||
$failedBuilds | ForEach-Object -Process { Write-Host "##vso[task.logissue type=error;]Name: $($_.Name); Link: $($_.Link)" }
|
||||
Write-Host "##vso[task.complete result=Failed]"
|
||||
} else {
|
||||
Write-host "##[section] All builds have been passed successfully"
|
||||
}
|
||||
}
|
||||
|
||||
$azureDevOpsApi = Get-AzureDevOpsApi -TeamFoundationCollectionUri $TeamFoundationCollectionUri `
|
||||
-ProjectName $AzureDevOpsProjectName `
|
||||
-AccessToken $AzureDevOpsAccessToken
|
||||
|
||||
$queuedBuilds = Queue-Builds -AzureDevOpsApi $azureDevOpsApi `
|
||||
-ToolVersions $ToolVersions `
|
||||
-SourceBranch $SourceBranch `
|
||||
-SourceVersion $SourceVersion `
|
||||
-DefinitionId $DefinitionId
|
||||
|
||||
Write-Host "Waiting results of builds ..."
|
||||
Wait-Builds -Builds $queuedBuilds
|
||||
|
||||
Make-BuildsOutput -Builds $queuedBuilds
|
||||
80
helpers/common-helpers.psm1
Normal file
80
helpers/common-helpers.psm1
Normal file
@ -0,0 +1,80 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
The execute command and print all output to the logs
|
||||
#>
|
||||
function Execute-Command {
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory=$true)][string] $Command
|
||||
)
|
||||
|
||||
Write-Debug "Execute $Command"
|
||||
|
||||
try {
|
||||
Invoke-Expression $Command | ForEach-Object { Write-Host $_ }
|
||||
}
|
||||
catch {
|
||||
Write-Host "Error happened during command execution: $Command"
|
||||
Write-Host "##vso[task.logissue type=error;] $_"
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Download file from url and return local path to file
|
||||
#>
|
||||
function Download-File {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[Uri]$Uri,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$OutputFolder
|
||||
)
|
||||
|
||||
$targetFilename = [IO.Path]::GetFileName($Uri)
|
||||
$targetFilepath = Join-Path $OutputFolder $targetFilename
|
||||
|
||||
Write-Debug "Download source from $Uri to $OutFile"
|
||||
try {
|
||||
(New-Object System.Net.WebClient).DownloadFile($Uri, $targetFilepath)
|
||||
return $targetFilepath
|
||||
} catch {
|
||||
Write-Host "Error during downloading file from '$Uri'"
|
||||
"$_"
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Generate file that contains the list of all files in particular directory
|
||||
#>
|
||||
function New-ToolStructureDump {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$ToolPath,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$OutputFolder
|
||||
)
|
||||
|
||||
$outputFile = Join-Path $OutputFolder "tools_structure.txt"
|
||||
|
||||
$folderContent = Get-ChildItem -Path $ToolPath -Recurse | Sort-Object | Select-Object -Property FullName, Length
|
||||
$folderContent | ForEach-Object {
|
||||
$relativePath = $_.FullName.Replace($ToolPath, "");
|
||||
return "${relativePath}"
|
||||
} | Out-File -FilePath $outputFile
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Check if it is macOS / Ubuntu platform
|
||||
#>
|
||||
function IsNixPlatform {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)] [ValidateNotNullOrEmpty()]
|
||||
[String]$Platform
|
||||
)
|
||||
|
||||
return ($Platform -match "macos") -or ($Platform -match "ubuntu")
|
||||
}
|
||||
158
helpers/generate-versions-manifest.ps1
Normal file
158
helpers/generate-versions-manifest.ps1
Normal file
@ -0,0 +1,158 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Generate versions manifest based on repository releases
|
||||
|
||||
.DESCRIPTION
|
||||
Versions manifest is needed to find the latest assets for particular version of tool
|
||||
.PARAMETER GitHubRepositoryOwner
|
||||
Required parameter. The organization which tool repository belongs
|
||||
.PARAMETER GitHubRepositoryName
|
||||
Optional parameter. The name of tool repository
|
||||
.PARAMETER GitHubAccessToken
|
||||
Required parameter. PAT Token to overcome GitHub API Rate limit
|
||||
.PARAMETER OutputFile
|
||||
Required parameter. File "*.json" where generated results will be saved
|
||||
.PARAMETER PlatformMapFile
|
||||
Optional parameter. Path to the json file with platform map
|
||||
Structure example:
|
||||
{
|
||||
"macos-1014": [
|
||||
{
|
||||
"platform": "darwin",
|
||||
"platform_version": "10.14"
|
||||
}, ...
|
||||
], ...
|
||||
}
|
||||
#>
|
||||
|
||||
param (
|
||||
[Parameter(Mandatory)] [string] $GitHubRepositoryOwner,
|
||||
[Parameter(Mandatory)] [string] $GitHubRepositoryName,
|
||||
[Parameter(Mandatory)] [string] $GitHubAccessToken,
|
||||
[Parameter(Mandatory)] [string] $OutputFile,
|
||||
[string] $PlatformMapFile
|
||||
)
|
||||
|
||||
Import-Module (Join-Path $PSScriptRoot "github/github-api.psm1")
|
||||
|
||||
if ($PlatformMapFile -and (Test-Path $PlatformMapFile)) {
|
||||
$PlatformMap = Get-Content $PlatformMapFile -Raw | ConvertFrom-Json -AsHashtable
|
||||
} else {
|
||||
$PlatformMap = @{}
|
||||
}
|
||||
|
||||
function Get-FileNameWithoutExtension {
|
||||
param (
|
||||
[Parameter(Mandatory)][string]$Filename
|
||||
)
|
||||
|
||||
if ($Filename.EndsWith(".tar.gz")) {
|
||||
$Filename = [IO.path]::GetFileNameWithoutExtension($Filename)
|
||||
}
|
||||
|
||||
return [IO.path]::GetFileNameWithoutExtension($Filename)
|
||||
}
|
||||
|
||||
function New-AssetItem {
|
||||
param (
|
||||
[Parameter(Mandatory)][string]$Filename,
|
||||
[Parameter(Mandatory)][string]$DownloadUrl,
|
||||
[Parameter(Mandatory)][string]$Arch,
|
||||
[Parameter(Mandatory)][string]$Platform,
|
||||
[string]$PlatformVersion
|
||||
)
|
||||
$asset = New-Object PSObject
|
||||
|
||||
$asset | Add-Member -Name "filename" -Value $Filename -MemberType NoteProperty
|
||||
$asset | Add-Member -Name "arch" -Value $Arch -MemberType NoteProperty
|
||||
$asset | Add-Member -Name "platform" -Value $Platform -MemberType NoteProperty
|
||||
if ($PlatformVersion) { $asset | Add-Member -Name "platform_version" -Value $PlatformVersion -MemberType NoteProperty }
|
||||
$asset | Add-Member -Name "download_url" -Value $DownloadUrl -MemberType NoteProperty
|
||||
|
||||
return $asset
|
||||
}
|
||||
|
||||
function Build-AssetsList {
|
||||
param (
|
||||
[AllowEmptyCollection()]
|
||||
[Parameter(Mandatory)][array]$ReleaseAssets
|
||||
)
|
||||
|
||||
|
||||
$assets = @()
|
||||
foreach($releaseAsset in $ReleaseAssets) {
|
||||
$filename = Get-FileNameWithoutExtension -Filename $releaseAsset.name
|
||||
$parts = $filename.Split("-")
|
||||
$arch = $parts[-1]
|
||||
$buildPlatform = [string]::Join("-", $parts[2..($parts.Length-2)])
|
||||
|
||||
if ($PlatformMap[$buildPlatform]) {
|
||||
$PlatformMap[$buildPlatform] | ForEach-Object {
|
||||
$assets += New-AssetItem -Filename $releaseAsset.name `
|
||||
-DownloadUrl $releaseAsset.browser_download_url `
|
||||
-Arch $arch `
|
||||
-Platform $_.platform `
|
||||
-PlatformVersion $_.platform_version
|
||||
}
|
||||
|
||||
} else {
|
||||
$assets += New-AssetItem -Filename $releaseAsset.name `
|
||||
-DownloadUrl $releaseAsset.browser_download_url `
|
||||
-Arch $arch `
|
||||
-Platform $buildPlatform
|
||||
}
|
||||
}
|
||||
|
||||
return $assets
|
||||
}
|
||||
|
||||
function Get-VersionFromRelease {
|
||||
param (
|
||||
[Parameter(Mandatory)][object]$Release
|
||||
)
|
||||
# Release name can contain additional information after ':' so filter it
|
||||
[string]$releaseName = $Release.name.Split(':')[0]
|
||||
[Version]$version = $null
|
||||
if (![Version]::TryParse($releaseName, [ref]$version)) {
|
||||
throw "Release '$($Release.id)' has invalid title '$($Release.name)'. It can't be parsed as version. ( $($Release.html_url) )"
|
||||
}
|
||||
|
||||
return $version
|
||||
}
|
||||
|
||||
function Build-VersionsManifest {
|
||||
param (
|
||||
[Parameter(Mandatory)][array]$Releases
|
||||
)
|
||||
|
||||
$Releases = $Releases | Sort-Object -Property "published_at" -Descending
|
||||
|
||||
$versionsHash = @{}
|
||||
foreach ($release in $Releases) {
|
||||
if (($release.draft -eq $true) -or ($release.prerelease -eq $true)) {
|
||||
continue
|
||||
}
|
||||
|
||||
[Version]$version = Get-VersionFromRelease $release
|
||||
$versionKey = $version.ToString()
|
||||
|
||||
if ($versionsHash.ContainsKey($versionKey)) {
|
||||
continue
|
||||
}
|
||||
|
||||
$versionsHash.Add($versionKey, [PSCustomObject]@{
|
||||
version = $versionKey
|
||||
stable = $true
|
||||
release_url = $release.html_url
|
||||
files = Build-AssetsList $release.assets
|
||||
})
|
||||
}
|
||||
|
||||
# Sort versions by descending
|
||||
return $versionsHash.Values | Sort-Object -Property @{ Expression = { [Version]$_.version }; Descending = $true }
|
||||
}
|
||||
|
||||
$gitHubApi = Get-GitHubApi -AccountName $GitHubRepositoryOwner -ProjectName $GitHubRepositoryName -AccessToken $GitHubAccessToken
|
||||
$releases = $gitHubApi.GetGitHubReleases()
|
||||
$versionIndex = Build-VersionsManifest $releases
|
||||
$versionIndex | ConvertTo-Json -Depth 5 | Out-File $OutputFile -Encoding UTF8NoBOM -Force
|
||||
106
helpers/github/create-pull-request.ps1
Normal file
106
helpers/github/create-pull-request.ps1
Normal file
@ -0,0 +1,106 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create commit with all unstaged changes in repository and create pull-request
|
||||
|
||||
.PARAMETER RepositoryOwner
|
||||
Required parameter. The organization which tool repository belongs
|
||||
.PARAMETER RepositoryName
|
||||
Optional parameter. The name of tool repository
|
||||
.PARAMETER AccessToken
|
||||
Required parameter. PAT Token to authorize
|
||||
.PARAMETER BranchName
|
||||
Required parameter. The name of branch where changes will be pushed
|
||||
.PARAMETER CommitMessage
|
||||
Required parameter. The commit message to push changes
|
||||
.PARAMETER PullRequestTitle
|
||||
Required parameter. The title of pull-request
|
||||
.PARAMETER PullRequestBody
|
||||
Required parameter. The description of pull-request
|
||||
#>
|
||||
param (
|
||||
[Parameter(Mandatory)] [string] $RepositoryOwner,
|
||||
[Parameter(Mandatory)] [string] $RepositoryName,
|
||||
[Parameter(Mandatory)] [string] $AccessToken,
|
||||
[Parameter(Mandatory)] [string] $BranchName,
|
||||
[Parameter(Mandatory)] [string] $CommitMessage,
|
||||
[Parameter(Mandatory)] [string] $PullRequestTitle,
|
||||
[Parameter(Mandatory)] [string] $PullRequestBody
|
||||
)
|
||||
|
||||
Import-Module (Join-Path $PSScriptRoot "github-api.psm1")
|
||||
Import-Module (Join-Path $PSScriptRoot "git.psm1")
|
||||
|
||||
function Update-PullRequest {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[object] $GitHubApi,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Title,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Body,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $BranchName,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[object] $PullRequest
|
||||
)
|
||||
|
||||
$updatedPullRequest = $GitHubApi.UpdatePullRequest($Title, $Body, $BranchName, $PullRequest.number)
|
||||
|
||||
if (($updatedPullRequest -eq $null) -or ($updatedPullRequest.html_url -eq $null)) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected error occurs while updating pull request."
|
||||
exit 1
|
||||
}
|
||||
Write-host "##[section] Pull request updated: $($updatedPullRequest.html_url)"
|
||||
}
|
||||
|
||||
function Create-PullRequest {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[object] $GitHubApi,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Title,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Body,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $BranchName
|
||||
)
|
||||
|
||||
$createdPullRequest = $GitHubApi.CreateNewPullRequest($Title, $Body, $BranchName)
|
||||
|
||||
if (($createdPullRequest -eq $null) -or ($createdPullRequest.html_url -eq $null)) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected error occurs while creating pull request."
|
||||
exit 1
|
||||
}
|
||||
|
||||
Write-host "##[section] Pull request created: $($createdPullRequest.html_url)"
|
||||
}
|
||||
|
||||
Write-Host "Configure local git preferences"
|
||||
Git-ConfigureUser -Name "Service account" -Email "no-reply@microsoft.com"
|
||||
|
||||
Write-Host "Create branch: $BranchName"
|
||||
Git-CreateBranch -Name $BranchName
|
||||
|
||||
Write-Host "Create commit"
|
||||
Git-CommitAllChanges -Message $CommitMessage
|
||||
|
||||
Write-Host "Push branch: $BranchName"
|
||||
Git-PushBranch -Name $BranchName -Force $true
|
||||
|
||||
$gitHubApi = Get-GitHubApi -AccountName $RepositoryOwner -ProjectName $RepositoryName -AccessToken $AccessToken
|
||||
$pullRequest = $gitHubApi.GetPullRequest($BranchName, $RepositoryOwner)
|
||||
|
||||
if ($pullRequest.Count -gt 0) {
|
||||
Write-Host "Update pull request"
|
||||
Update-PullRequest -GitHubApi $gitHubApi `
|
||||
-Title $PullRequestTitle `
|
||||
-Body $PullRequestBody `
|
||||
-BranchName $BranchName `
|
||||
-PullRequest $pullRequest[0]
|
||||
} else {
|
||||
Write-Host "Create pull request"
|
||||
Create-PullRequest -GitHubApi $gitHubApi `
|
||||
-Title $PullRequestTitle `
|
||||
-Body $PullRequestBody `
|
||||
-BranchName $BranchName
|
||||
}
|
||||
81
helpers/github/git.psm1
Normal file
81
helpers/github/git.psm1
Normal file
@ -0,0 +1,81 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Configure git credentials to use with commits
|
||||
#>
|
||||
function Git-ConfigureUser {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Name,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Email
|
||||
)
|
||||
|
||||
git config --global user.name $Name | Out-Host
|
||||
git config --global user.email $Email | Out-Host
|
||||
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected failure occurs while configuring git preferences."
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create new branch
|
||||
#>
|
||||
function Git-CreateBranch {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Name
|
||||
)
|
||||
|
||||
git checkout -b $Name | Out-Host
|
||||
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected failure occurs while creating new branch: $Name."
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Commit all staged and unstaged changes
|
||||
#>
|
||||
function Git-CommitAllChanges {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Message
|
||||
)
|
||||
|
||||
git add -A | Out-Host
|
||||
git commit -m "$Message" | Out-Host
|
||||
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected failure occurs while commiting changes."
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Push branch to remote repository
|
||||
#>
|
||||
function Git-PushBranch {
|
||||
Param (
|
||||
[Parameter(Mandatory=$true)]
|
||||
[string] $Name,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[boolean] $Force
|
||||
)
|
||||
|
||||
if ($Force) {
|
||||
git push --set-upstream origin $Name --force | Out-Host
|
||||
} else {
|
||||
git push --set-upstream origin $Name | Out-Host
|
||||
}
|
||||
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host "##vso[task.logissue type=error;] Unexpected failure occurs while pushing changes."
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
109
helpers/github/github-api.psm1
Normal file
109
helpers/github/github-api.psm1
Normal file
@ -0,0 +1,109 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
The module that contains a bunch of methods to interact with GitHub API V3
|
||||
#>
|
||||
class GitHubApi
|
||||
{
|
||||
[string] $BaseUrl
|
||||
[string] $RepoOwner
|
||||
[object] $AuthHeader
|
||||
|
||||
GitHubApi(
|
||||
[string] $AccountName,
|
||||
[string] $ProjectName,
|
||||
[string] $AccessToken
|
||||
) {
|
||||
$this.BaseUrl = $this.BuildBaseUrl($AccountName, $ProjectName)
|
||||
$this.AuthHeader = $this.BuildAuth($AccessToken)
|
||||
}
|
||||
|
||||
[object] hidden BuildAuth([string]$AccessToken) {
|
||||
if ([string]::IsNullOrEmpty($AccessToken)) {
|
||||
return $null
|
||||
}
|
||||
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("'':${AccessToken}"))
|
||||
return @{
|
||||
Authorization = "Basic ${base64AuthInfo}"
|
||||
}
|
||||
}
|
||||
|
||||
[string] hidden BuildBaseUrl([string]$RepositoryOwner, [string]$RepositoryName) {
|
||||
return "https://api.github.com/repos/$RepositoryOwner/$RepositoryName"
|
||||
}
|
||||
|
||||
[object] CreateNewPullRequest([string]$Title, [string]$Body, [string]$BranchName){
|
||||
$requestBody = @{
|
||||
title = $Title
|
||||
body = $Body
|
||||
head = $BranchName
|
||||
base = "master"
|
||||
} | ConvertTo-Json
|
||||
|
||||
$url = "pulls"
|
||||
return $this.InvokeRestMethod($url, 'Post', $null, $requestBody)
|
||||
}
|
||||
|
||||
[object] GetPullRequest([string]$BranchName, [string]$RepositoryOwner){
|
||||
$url = "pulls"
|
||||
return $this.InvokeRestMethod($url, 'GET', "head=${RepositoryOwner}:$BranchName&base=master", $null)
|
||||
}
|
||||
|
||||
[object] UpdatePullRequest([string]$Title, [string]$Body, [string]$BranchName, [string]$PullRequestNumber){
|
||||
$requestBody = @{
|
||||
title = $Title
|
||||
body = $Body
|
||||
head = $BranchName
|
||||
base = "master"
|
||||
} | ConvertTo-Json
|
||||
|
||||
$url = "pulls/$PullRequestNumber"
|
||||
return $this.InvokeRestMethod($url, 'Post', $null, $requestBody)
|
||||
}
|
||||
|
||||
[object] GetGitHubReleases(){
|
||||
$url = "releases"
|
||||
return $this.InvokeRestMethod($url, 'GET', $null, $null)
|
||||
}
|
||||
|
||||
[string] hidden BuildUrl([string]$Url, [string]$RequestParams) {
|
||||
if ([string]::IsNullOrEmpty($RequestParams)) {
|
||||
return "$($this.BaseUrl)/$($Url)"
|
||||
} else {
|
||||
return "$($this.BaseUrl)/$($Url)?$($RequestParams)"
|
||||
}
|
||||
}
|
||||
|
||||
[object] hidden InvokeRestMethod(
|
||||
[string] $Url,
|
||||
[string] $Method,
|
||||
[string] $RequestParams,
|
||||
[string] $Body
|
||||
) {
|
||||
$requestUrl = $this.BuildUrl($Url, $RequestParams)
|
||||
$params = @{
|
||||
Method = $Method
|
||||
ContentType = "application/json"
|
||||
Uri = $requestUrl
|
||||
Headers = @{}
|
||||
}
|
||||
if ($this.AuthHeader) {
|
||||
$params.Headers += $this.AuthHeader
|
||||
}
|
||||
if (![string]::IsNullOrEmpty($Body)) {
|
||||
$params.Body = $Body
|
||||
}
|
||||
|
||||
return Invoke-RestMethod @params
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
function Get-GitHubApi {
|
||||
param (
|
||||
[string] $AccountName,
|
||||
[string] $ProjectName,
|
||||
[string] $AccessToken
|
||||
)
|
||||
|
||||
return [GitHubApi]::New($AccountName, $ProjectName, $AccessToken)
|
||||
}
|
||||
50
helpers/nix-helpers.psm1
Normal file
50
helpers/nix-helpers.psm1
Normal file
@ -0,0 +1,50 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Pack folder to *.zip format
|
||||
#>
|
||||
function Pack-Zip {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$PathToArchive,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$ToolZipFile
|
||||
)
|
||||
|
||||
Write-Debug "Pack $PathToArchive to $ToolZipFile"
|
||||
Push-Location -Path $PathToArchive
|
||||
zip -q -r $ToolZipFile * | Out-Null
|
||||
Pop-Location
|
||||
}
|
||||
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Unpack *.tar file
|
||||
#>
|
||||
function Extract-TarArchive {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$ArchivePath,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$OutputDirectory
|
||||
)
|
||||
|
||||
Write-Debug "tar -C $OutputDirectory -xzf $ArchivePath --strip 1"
|
||||
tar -C $OutputDirectory -xzf $ArchivePath --strip 1
|
||||
}
|
||||
|
||||
function Create-TarArchive {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$SourceFolder,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$ArchivePath,
|
||||
[string]$CompressionType = "gz"
|
||||
)
|
||||
|
||||
$CompressionTypeArgument = If ([string]::IsNullOrWhiteSpace($CompressionType)) { "" } else { "--${CompressionType}" }
|
||||
|
||||
Push-Location $SourceFolder
|
||||
Write-Debug "tar -c $CompressionTypeArgument -f $ArchivePath ."
|
||||
tar -c $CompressionTypeArgument -f $ArchivePath .
|
||||
Pop-Location
|
||||
}
|
||||
33
helpers/pester-extensions.psm1
Normal file
33
helpers/pester-extensions.psm1
Normal file
@ -0,0 +1,33 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Pester extension that allows to run command and validate exit code
|
||||
.EXAMPLE
|
||||
"python file.py" | Should -ReturnZeroExitCode
|
||||
#>
|
||||
function ShouldReturnZeroExitCode {
|
||||
Param(
|
||||
[Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()]
|
||||
[String]$ActualValue,
|
||||
[switch]$Negate
|
||||
)
|
||||
|
||||
Write-Host "Run command '${ActualValue}'"
|
||||
Invoke-Expression -Command $ActualValue | ForEach-Object { Write-Host $_ }
|
||||
$actualExitCode = $LASTEXITCODE
|
||||
|
||||
[bool]$succeeded = $actualExitCode -eq 0
|
||||
if ($Negate) { $succeeded = -not $succeeded }
|
||||
|
||||
if (-not $succeeded)
|
||||
{
|
||||
$failureMessage = "Command '${ActualValue}' has finished with exit code ${actualExitCode}"
|
||||
}
|
||||
|
||||
return New-Object PSObject -Property @{
|
||||
Succeeded = $succeeded
|
||||
FailureMessage = $failureMessage
|
||||
}
|
||||
}
|
||||
|
||||
Add-AssertionOperator -Name ReturnZeroExitCode `
|
||||
-Test $function:ShouldReturnZeroExitCode
|
||||
18
helpers/win-helpers.psm1
Normal file
18
helpers/win-helpers.psm1
Normal file
@ -0,0 +1,18 @@
|
||||
function Create-SevenZipArchive {
|
||||
param(
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$SourceFolder,
|
||||
[Parameter(Mandatory=$true)]
|
||||
[String]$ArchivePath,
|
||||
[String]$ArchiveType = "zip",
|
||||
[String]$CompressionLevel = 5
|
||||
)
|
||||
|
||||
$ArchiveTypeArgument = "-t${ArchiveType}"
|
||||
$CompressionLevelArgument = "-mx=${CompressionLevel}"
|
||||
|
||||
Push-Location $SourceFolder
|
||||
Write-Debug "7z a $ArchiveTypeArgument $CompressionLevelArgument $ArchivePath @$SourceFolder"
|
||||
7z a $ArchiveTypeArgument $CompressionLevelArgument $ArchivePath $SourceFolder\*
|
||||
Pop-Location
|
||||
}
|
||||
29
installers/map_python_platforms.json
Normal file
29
installers/map_python_platforms.json
Normal file
@ -0,0 +1,29 @@
|
||||
{
|
||||
"macos-1014": [
|
||||
{
|
||||
"platform": "darwin",
|
||||
"platform_version": "10.14"
|
||||
},
|
||||
{
|
||||
"platform": "darwin",
|
||||
"platform_version": "10.15"
|
||||
}
|
||||
],
|
||||
"ubuntu-1604": [
|
||||
{
|
||||
"platform": "linux",
|
||||
"platform_version": "16.04"
|
||||
}
|
||||
],
|
||||
"ubuntu-1804": [
|
||||
{
|
||||
"platform": "linux",
|
||||
"platform_version": "18.04"
|
||||
}
|
||||
],
|
||||
"windows-2016": [
|
||||
{
|
||||
"platform": "win32"
|
||||
}
|
||||
]
|
||||
}
|
||||
50
installers/nix-setup-template.sh
Normal file
50
installers/nix-setup-template.sh
Normal file
@ -0,0 +1,50 @@
|
||||
set -e
|
||||
|
||||
MAJOR_VERSION={0}
|
||||
MINOR_VERSION={1}
|
||||
BUILD_VERSION={2}
|
||||
|
||||
PYTHON_MAJOR=python$MAJOR_VERSION
|
||||
PYTHON_MAJOR_DOT_MINOR=python$MAJOR_VERSION.$MINOR_VERSION
|
||||
PYTHON_MAJORMINOR=python$MAJOR_VERSION$MINOR_VERSION
|
||||
PYTHON_FULL_VERSION=$MAJOR_VERSION.$MINOR_VERSION.$BUILD_VERSION
|
||||
|
||||
PYTHON_TOOLCACHE_PATH=$AGENT_TOOLSDIRECTORY/Python
|
||||
PYTHON_TOOLCACHE_VERSION_PATH=$PYTHON_TOOLCACHE_PATH/$PYTHON_FULL_VERSION
|
||||
PYTHON_TOOLCACHE_VERSION_ARCH_PATH=$PYTHON_TOOLCACHE_VERSION_PATH/x64
|
||||
|
||||
echo "Check if Python hostedtoolcache folder exist..."
|
||||
if [ ! -d $PYTHON_TOOLCACHE_PATH ]; then
|
||||
echo "Creating Python hostedtoolcache folder..."
|
||||
mkdir -p $PYTHON_TOOLCACHE_PATH
|
||||
elif [ -d $PYTHON_TOOLCACHE_VERSION_PATH ]; then
|
||||
echo "Deleting Python $PYTHON_FULL_VERSION"
|
||||
rm -rf $PYTHON_TOOLCACHE_VERSION_PATH
|
||||
fi
|
||||
|
||||
echo "Create Python $PYTHON_FULL_VERSION folder"
|
||||
mkdir -p $PYTHON_TOOLCACHE_VERSION_ARCH_PATH
|
||||
|
||||
echo "Copy Python binaries to hostedtoolcache folder"
|
||||
cp -R ./* $PYTHON_TOOLCACHE_VERSION_ARCH_PATH
|
||||
rm $PYTHON_TOOLCACHE_VERSION_ARCH_PATH/setup.sh
|
||||
|
||||
cd $PYTHON_TOOLCACHE_VERSION_ARCH_PATH
|
||||
|
||||
echo "Create additional symlinks (Required for UsePythonVersion VSTS task)"
|
||||
ln -s ./bin/$PYTHON_MAJOR_DOT_MINOR python
|
||||
|
||||
cd bin/
|
||||
ln -s $PYTHON_MAJOR_DOT_MINOR $PYTHON_MAJORMINOR
|
||||
if [ ! -f python ]; then
|
||||
ln -s $PYTHON_MAJOR_DOT_MINOR python
|
||||
fi
|
||||
|
||||
chmod +x ../python $PYTHON_MAJOR $PYTHON_MAJOR_DOT_MINOR $PYTHON_MAJORMINOR python
|
||||
|
||||
echo "Upgrading PIP..."
|
||||
./python -m ensurepip
|
||||
./python -m pip install --ignore-installed pip
|
||||
|
||||
echo "Create complete file"
|
||||
touch $PYTHON_TOOLCACHE_VERSION_PATH/x64.complete
|
||||
138
installers/win-setup-template.ps1
Normal file
138
installers/win-setup-template.ps1
Normal file
@ -0,0 +1,138 @@
|
||||
[String] $Architecture = "{{__ARCHITECTURE__}}"
|
||||
[Version] $Version = "{{__VERSION__}}"
|
||||
[String] $PythonExecName = "{{__PYTHON_EXEC_NAME__}}"
|
||||
|
||||
function Get-RegistryVersionFilter {
|
||||
param
|
||||
(
|
||||
[Parameter(Mandatory)][String] $Architecture,
|
||||
[Parameter(Mandatory)][Int32] $MajorVersion,
|
||||
[Parameter(Mandatory)][Int32] $MinorVersion
|
||||
)
|
||||
|
||||
$archFilter = if ($Architecture -eq 'x86') { "32-bit" } else { "64-bit" }
|
||||
### Python 2.7 x86 have no architecture postfix
|
||||
if (($Architecture -eq "x86") -and ($MajorVersion -eq 2))
|
||||
{
|
||||
"Python $MajorVersion.$MinorVersion.\d+$"
|
||||
}
|
||||
else
|
||||
{
|
||||
"Python $MajorVersion.$MinorVersion.*($archFilter)"
|
||||
}
|
||||
}
|
||||
|
||||
function Remove-RegistryEntries
|
||||
{
|
||||
param
|
||||
(
|
||||
[Parameter(Mandatory)][String] $Architecture,
|
||||
[Parameter(Mandatory)][Int32] $MajorVersion,
|
||||
[Parameter(Mandatory)][Int32] $MinorVersion
|
||||
)
|
||||
|
||||
$versionFilter = Get-RegistryVersionFilter -Architecture $Architecture -MajorVersion $MajorVersion -MinorVersion $MinorVersion
|
||||
|
||||
$regPath = "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Products"
|
||||
$regKeys = Get-ChildItem -Path Registry::$regPath -Recurse | Where-Object Property -Ccontains DisplayName
|
||||
foreach ($key in $regKeys)
|
||||
{
|
||||
if ($key.getValue("DisplayName") -match $versionFilter)
|
||||
{
|
||||
Remove-Item -Path $key.PSParentPath -Recurse -Force -Verbose
|
||||
}
|
||||
}
|
||||
|
||||
$regPath = "HKEY_CLASSES_ROOT\Installer\Products"
|
||||
Get-ChildItem -Path Registry::$regPath | Where-Object { $_.GetValue("ProductName") -match $versionFilter } | ForEach-Object {
|
||||
Remove-Item Registry::$_ -Recurse -Force -Verbose
|
||||
}
|
||||
|
||||
$uninstallRegistrySections = @(
|
||||
"HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Uninstall", # current user, x64
|
||||
"HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Uninstall", # all users, x64
|
||||
"HKEY_CURRENT_USER\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall", # current user, x86
|
||||
"HKEY_LOCAL_MACHINE\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall" # all users, x86
|
||||
)
|
||||
|
||||
$uninstallRegistrySections | Where-Object { Test-Path -Path Registry::$_ } | ForEach-Object {
|
||||
Get-ChildItem -Path Registry::$_ | Where-Object { $_.getValue("DisplayName") -match $versionFilter } | ForEach-Object {
|
||||
Remove-Item Registry::$_ -Recurse -Force -Verbose
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Get-ExecParams {
|
||||
param
|
||||
(
|
||||
[Parameter(Mandatory)][Boolean] $IsMSI,
|
||||
[Parameter(Mandatory)][String] $PythonArchPath
|
||||
)
|
||||
|
||||
if ($IsMSI)
|
||||
{
|
||||
"TARGETDIR=$PythonArchPath ALLUSERS=1"
|
||||
}
|
||||
else
|
||||
{
|
||||
"DefaultAllUsersTargetDir=$PythonArchPath InstallAllUsers=1"
|
||||
}
|
||||
}
|
||||
|
||||
$ToolcacheRoot = $env:AGENT_TOOLSDIRECTORY
|
||||
if ([string]::IsNullOrEmpty($ToolcacheRoot))
|
||||
{
|
||||
# GitHub images don't have `AGENT_TOOLSDIRECTORY` variable
|
||||
$ToolcacheRoot = $env:RUNNER_TOOL_CACHE
|
||||
}
|
||||
$PythonToolcachePath = Join-Path -Path $ToolcacheRoot -ChildPath "Python"
|
||||
$PythonVersionPath = Join-Path -Path $PythonToolcachePath -ChildPath $Version.ToString()
|
||||
$PythonArchPath = Join-Path -Path $PythonVersionPath -ChildPath $Architecture
|
||||
|
||||
$IsMSI = $PythonExecName -match "msi"
|
||||
|
||||
$MajorVersion = $Version.Major
|
||||
$MinorVersion = $Version.Minor
|
||||
|
||||
Write-Host "Check if Python hostedtoolcache folder exist..."
|
||||
if (-Not (Test-Path $PythonToolcachePath))
|
||||
{
|
||||
Write-Host "Create Python toolcache folder"
|
||||
New-Item -ItemType Directory -Path $PythonToolcachePath | Out-Null
|
||||
}
|
||||
|
||||
Write-Host "Check if current Python version is installed..."
|
||||
$InstalledVersion = Get-ChildItem -Path $PythonToolcachePath -Filter "$MajorVersion.$MinorVersion.*"
|
||||
|
||||
Write-Host "Remove registry entries for Python ${MajorVersion}.${MinorVersion}(${Architecture})..."
|
||||
Remove-RegistryEntries -Architecture $Architecture -MajorVersion $MajorVersion -MinorVersion $MinorVersion
|
||||
|
||||
if (($null -ne $InstalledVersion) -and (Test-Path -Path $InstalledVersion.FullName)) {
|
||||
Write-Host "Python$MajorVersion.$MinorVersion was found in $PythonToolcachePath"
|
||||
Write-Host "Deleting $($InstalledVersion.FullName)..."
|
||||
Remove-Item -Path $InstalledVersion.FullName -Recurse -Force
|
||||
} else {
|
||||
Write-Host "No Python$MajorVersion.$MinorVersion.* found"
|
||||
}
|
||||
|
||||
Write-Host "Create Python $Version folder in $PythonToolcachePath"
|
||||
New-Item -ItemType Directory -Path $PythonArchPath -Force | Out-Null
|
||||
|
||||
Write-Host "Copy Python binaries to $PythonArchPath"
|
||||
Copy-Item -Path ./$PythonExecName -Destination $PythonArchPath | Out-Null
|
||||
|
||||
Write-Host "Install Python $Version in $PythonToolcachePath..."
|
||||
$ExecParams = Get-ExecParams -IsMSI $IsMSI -PythonArchPath $PythonArchPath
|
||||
|
||||
cmd.exe /c "cd $PythonArchPath && call $PythonExecName $ExecParams /quiet"
|
||||
if ($LASTEXITCODE -ne 0)
|
||||
{
|
||||
Throw "Error happened during Python installation"
|
||||
}
|
||||
|
||||
Write-Host "Install and upgrade Pip"
|
||||
$PythonExePath = Join-Path -Path $PythonArchPath -ChildPath "python.exe"
|
||||
cmd.exe /c "$PythonExePath -m ensurepip && $PythonExePath -m pip install --upgrade pip"
|
||||
|
||||
Write-Host "Create complete file"
|
||||
New-Item -ItemType File -Path $PythonVersionPath -Name "$Architecture.complete" | Out-Null
|
||||
11
tests/clean-toolcache.ps1
Normal file
11
tests/clean-toolcache.ps1
Normal file
@ -0,0 +1,11 @@
|
||||
if ($env:PLATFORM -match 'windows') {
|
||||
$PythonFilter = "Name like '%Python%'"
|
||||
Get-WmiObject Win32_Product -Filter $PythonFilter | Foreach-Object {
|
||||
Write-Host "Uninstalling $($_.Name) ..."
|
||||
$_.Uninstall() | Out-Null
|
||||
}
|
||||
}
|
||||
|
||||
$PythonToolcachePath = Join-Path -Path $env:AGENT_TOOLSDIRECTORY -ChildPath "Python"
|
||||
Write-Host "Removing Python toolcache directory ..."
|
||||
Remove-Item -Path $PythonToolcachePath -Recurse -Force
|
||||
72
tests/python-tests.ps1
Normal file
72
tests/python-tests.ps1
Normal file
@ -0,0 +1,72 @@
|
||||
param (
|
||||
[Version] [Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()]
|
||||
$Version,
|
||||
[String] [Parameter (Mandatory = $true)] [ValidateNotNullOrEmpty()]
|
||||
$Platform
|
||||
)
|
||||
|
||||
Import-Module (Join-Path $PSScriptRoot "../helpers/pester-extensions.psm1")
|
||||
Import-Module (Join-Path $PSScriptRoot "../helpers/common-helpers.psm1")
|
||||
|
||||
function Analyze-MissingModules([string] $buildOutputLocation) {
|
||||
$searchStringStart = "Failed to build these modules:"
|
||||
$searchStringEnd = "running build_scripts"
|
||||
$pattern = "$searchStringStart(.*?)$searchStringEnd"
|
||||
|
||||
$buildContent = Get-Content -Path $buildOutputLocation
|
||||
$splitBuiltOutput = $buildContent -split "\n";
|
||||
|
||||
### Search for missing modules that are displayed between the search strings
|
||||
$regexMatch = [regex]::match($SplitBuiltOutput, $Pattern)
|
||||
if ($regexMatch.Success)
|
||||
{
|
||||
Write-Host "Failed missing modules:"
|
||||
Write-Host $regexMatch.Groups[1].Value
|
||||
return 1
|
||||
}
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
Describe "Tests" {
|
||||
It "Python version" {
|
||||
"python --version" | Should -ReturnZeroExitCode
|
||||
$pythonLocation = (Get-Command "python").Path
|
||||
$pythonLocation | Should -Not -BeNullOrEmpty
|
||||
$expectedPath = Join-Path -Path $env:AGENT_TOOLSDIRECTORY -ChildPath "Python"
|
||||
$pythonLocation.startsWith($expectedPath) | Should -BeTrue
|
||||
}
|
||||
|
||||
It "Run simple code" {
|
||||
"python ./sources/simple-test.py" | Should -ReturnZeroExitCode
|
||||
}
|
||||
|
||||
if (IsNixPlatform $Platform) {
|
||||
|
||||
It "Check for failed modules in build_output" {
|
||||
$buildOutputLocation = Join-Path $env:BUILD_BINARIESDIRECTORY "build_output.txt"
|
||||
Analyze-MissingModules $buildOutputLocation | Should -Be 0
|
||||
}
|
||||
|
||||
It "Check if all required python modules are installed" {
|
||||
"python ./sources/python-modules.py" | Should -ReturnZeroExitCode
|
||||
}
|
||||
|
||||
It "Check if python configuration is correct" {
|
||||
"python ./sources/python-config-test.py" | Should -ReturnZeroExitCode
|
||||
}
|
||||
|
||||
It "Check if shared libraries are linked correctly" {
|
||||
"bash ./sources/psutil-install-test.sh" | Should -ReturnZeroExitCode
|
||||
}
|
||||
}
|
||||
|
||||
# Pyinstaller 3.5 does not support Python 3.8.0. Check issue https://github.com/pyinstaller/pyinstaller/issues/4311
|
||||
if ($Version -lt "3.8.0") {
|
||||
It "Validate Pyinstaller" {
|
||||
"pip install pyinstaller" | Should -ReturnZeroExitCode
|
||||
"pyinstaller --onefile ./sources/simple-test.py" | Should -ReturnZeroExitCode
|
||||
"./dist/simple-test" | Should -ReturnZeroExitCode
|
||||
}
|
||||
}
|
||||
}
|
||||
5
tests/sources/psutil-install-test.sh
Normal file
5
tests/sources/psutil-install-test.sh
Normal file
@ -0,0 +1,5 @@
|
||||
# Check if shared libraries are linked correctly
|
||||
python -m venv /tmp/aml-ve
|
||||
source /tmp/aml-ve/bin/activate
|
||||
easy_install --version
|
||||
pip install psutil --verbose
|
||||
6
tests/sources/python-config-output.py
Normal file
6
tests/sources/python-config-output.py
Normal file
@ -0,0 +1,6 @@
|
||||
import distutils.sysconfig
|
||||
import sysconfig
|
||||
|
||||
from pprint import pprint
|
||||
pprint(sysconfig.get_config_vars())
|
||||
pprint(distutils.sysconfig.get_config_vars())
|
||||
68
tests/sources/python-config-test.py
Normal file
68
tests/sources/python-config-test.py
Normal file
@ -0,0 +1,68 @@
|
||||
import distutils.sysconfig
|
||||
import sysconfig
|
||||
import sys
|
||||
import platform
|
||||
import os
|
||||
|
||||
# Define variables
|
||||
os_type = platform.system()
|
||||
version = sys.version.split(" ")[0]
|
||||
|
||||
lib_dir_path = sysconfig.get_config_var('LIBDIR')
|
||||
ld_library_name = sysconfig.get_config_var('LDLIBRARY')
|
||||
|
||||
is_shared = sysconfig.get_config_var('Py_ENABLE_SHARED')
|
||||
have_libreadline = sysconfig.get_config_var("HAVE_LIBREADLINE")
|
||||
|
||||
### Define expected variables
|
||||
if os_type == 'Linux': expected_ld_library_extension = 'so'
|
||||
if os_type == 'Darwin': expected_ld_library_extension = 'dylib'
|
||||
expected_lib_dir_path = '{0}/Python/{1}/x64/lib'.format(os.getenv("AGENT_TOOLSDIRECTORY"), version)
|
||||
|
||||
# Check modules
|
||||
### Validate libraries path
|
||||
if lib_dir_path != expected_lib_dir_path:
|
||||
print('Invalid libraries location: %s; Expected: %s' % (lib_dir_path, expected_lib_dir_path))
|
||||
exit(1)
|
||||
|
||||
### Validate shared libraries
|
||||
if is_shared:
|
||||
print('%s was built with shared extensions' % ld_library_name)
|
||||
|
||||
### Validate libpython extension
|
||||
ld_library_extension = ld_library_name.split('.')[-1]
|
||||
if ld_library_extension != expected_ld_library_extension:
|
||||
print('Invalid extension: %s; Expected %s' % (ld_library_extension, expected_ld_library_extension))
|
||||
exit(1)
|
||||
else:
|
||||
print('%s was built without shared extensions' % ld_library_name)
|
||||
exit(1)
|
||||
|
||||
### Validate macOS
|
||||
if os_type == 'Darwin':
|
||||
### Validate openssl links
|
||||
if version < "3.7.0":
|
||||
expected_ldflags = '-L/usr/local/opt/openssl@1.1/lib'
|
||||
ldflags = sysconfig.get_config_var('LDFLAGS')
|
||||
|
||||
if not expected_ldflags in ldflags:
|
||||
print('Invalid ldflags: %s; Expected: %s' % (ldflags, expected_ldflags))
|
||||
exit(1)
|
||||
else:
|
||||
expected_openssl_includes = '-I/usr/local/opt/openssl/include'
|
||||
expected_openssl_ldflags ='-L/usr/local/opt/openssl/lib'
|
||||
|
||||
openssl_includes = sysconfig.get_config_var('OPENSSL_INCLUDES')
|
||||
openssl_ldflags = sysconfig.get_config_var('OPENSSL_LDFLAGS')
|
||||
|
||||
if openssl_includes != expected_openssl_includes:
|
||||
print('Invalid openssl_includes: %s; Expected: %s' % (openssl_includes, expected_openssl_includes))
|
||||
exit(1)
|
||||
if openssl_ldflags != expected_openssl_ldflags:
|
||||
print('Invalid openssl_ldflags: %s; Expected: %s' % (openssl_ldflags, expected_openssl_ldflags))
|
||||
exit(1)
|
||||
|
||||
### Validate libreadline
|
||||
if not have_libreadline:
|
||||
print('Missing libreadline')
|
||||
exit(1)
|
||||
276
tests/sources/python-modules.py
Normal file
276
tests/sources/python-modules.py
Normal file
@ -0,0 +1,276 @@
|
||||
"""
|
||||
Make sure all the optional modules are installed.
|
||||
This is needed for Linux since we build from source.
|
||||
"""
|
||||
|
||||
from __future__ import print_function
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
|
||||
# The Python standard library as of Python 3.0
|
||||
standard_library = [
|
||||
'abc',
|
||||
'aifc',
|
||||
'antigravity',
|
||||
'argparse',
|
||||
'ast',
|
||||
'asynchat',
|
||||
'asyncore',
|
||||
'base64',
|
||||
'bdb',
|
||||
'binhex',
|
||||
'bisect',
|
||||
'bz2',
|
||||
'cProfile',
|
||||
'calendar',
|
||||
'cgi',
|
||||
'cgitb',
|
||||
'chunk',
|
||||
'cmd',
|
||||
'code',
|
||||
'codecs',
|
||||
'codeop',
|
||||
'collections',
|
||||
'colorsys',
|
||||
'compileall',
|
||||
'configparser',
|
||||
'contextlib',
|
||||
'copy',
|
||||
'copyreg',
|
||||
'crypt',
|
||||
'csv',
|
||||
'ctypes',
|
||||
'curses',
|
||||
'datetime',
|
||||
'dbm',
|
||||
'decimal',
|
||||
'difflib',
|
||||
'dis',
|
||||
'distutils',
|
||||
'doctest',
|
||||
'dummy_threading',
|
||||
'email',
|
||||
'encodings',
|
||||
'filecmp',
|
||||
'fileinput',
|
||||
'fnmatch',
|
||||
'formatter',
|
||||
'fractions',
|
||||
'ftplib',
|
||||
'functools',
|
||||
'genericpath',
|
||||
'getopt',
|
||||
'getpass',
|
||||
'gettext',
|
||||
'glob',
|
||||
'gzip',
|
||||
'hashlib',
|
||||
'heapq',
|
||||
'hmac',
|
||||
'html',
|
||||
'http',
|
||||
'idlelib',
|
||||
'imaplib',
|
||||
'imghdr',
|
||||
'imp',
|
||||
'importlib',
|
||||
'inspect',
|
||||
'io',
|
||||
'json',
|
||||
'keyword',
|
||||
'lib2to3',
|
||||
'linecache',
|
||||
'locale',
|
||||
'logging',
|
||||
'macpath',
|
||||
'mailbox',
|
||||
'mailcap',
|
||||
'mimetypes',
|
||||
'modulefinder',
|
||||
'multiprocessing',
|
||||
'netrc',
|
||||
'nntplib',
|
||||
'ntpath',
|
||||
'nturl2path',
|
||||
'numbers',
|
||||
'opcode',
|
||||
'operator',
|
||||
'optparse',
|
||||
'os',
|
||||
'pdb',
|
||||
'pickle',
|
||||
'pickletools',
|
||||
'pipes',
|
||||
'pkgutil',
|
||||
'platform',
|
||||
'plistlib',
|
||||
'poplib',
|
||||
'posixpath',
|
||||
'pprint',
|
||||
'profile',
|
||||
'pstats',
|
||||
'pty',
|
||||
'py_compile',
|
||||
'pyclbr',
|
||||
'pydoc',
|
||||
'pydoc_data',
|
||||
'queue',
|
||||
'quopri',
|
||||
'random',
|
||||
're',
|
||||
'readline',
|
||||
'reprlib',
|
||||
'rlcompleter',
|
||||
'runpy',
|
||||
'sched',
|
||||
'shelve',
|
||||
'shlex',
|
||||
'shutil',
|
||||
'signal',
|
||||
'site',
|
||||
'smtpd',
|
||||
'smtplib',
|
||||
'sndhdr',
|
||||
'socket',
|
||||
'socketserver',
|
||||
'sqlite3',
|
||||
'sre_compile',
|
||||
'sre_constants',
|
||||
'sre_parse',
|
||||
'ssl',
|
||||
'stat',
|
||||
'string',
|
||||
'stringprep',
|
||||
'struct',
|
||||
'subprocess',
|
||||
'sunau',
|
||||
'symbol',
|
||||
'symtable',
|
||||
'sysconfig',
|
||||
'tabnanny',
|
||||
'tarfile',
|
||||
'telnetlib',
|
||||
'tempfile',
|
||||
'test',
|
||||
'textwrap',
|
||||
'this',
|
||||
'threading',
|
||||
'timeit',
|
||||
'tkinter',
|
||||
'token',
|
||||
'tokenize',
|
||||
'trace',
|
||||
'traceback',
|
||||
'tty',
|
||||
'turtle',
|
||||
'turtledemo',
|
||||
'types',
|
||||
'unittest',
|
||||
'urllib',
|
||||
'uu',
|
||||
'uuid',
|
||||
'warnings',
|
||||
'wave',
|
||||
'weakref',
|
||||
'webbrowser',
|
||||
'wsgiref',
|
||||
'xdrlib',
|
||||
'xml',
|
||||
'xmlrpc',
|
||||
'zipfile'
|
||||
]
|
||||
|
||||
# Modules that had different names in Python 2
|
||||
if sys.version_info.major == 2:
|
||||
def replace(lst, old, new):
|
||||
lst[lst.index(old)] = new
|
||||
|
||||
# Keys are the Python 2 names
|
||||
# Values are the Python 3 names
|
||||
renames = {
|
||||
'ConfigParser': 'configparser',
|
||||
'copy_reg': 'copyreg',
|
||||
'HTMLParser': 'html',
|
||||
'httplib': 'http',
|
||||
'Queue': 'queue',
|
||||
'repr': 'reprlib',
|
||||
'SocketServer': 'socketserver',
|
||||
'xmlrpclib': 'xmlrpc',
|
||||
'Tkinter': 'tkinter'
|
||||
}
|
||||
|
||||
# All of the Python 3 names should be in the list
|
||||
for python2name, python3name in renames.items():
|
||||
replace(standard_library, python3name, python2name)
|
||||
|
||||
# Add new modules
|
||||
# See https://docs.python.org/3/whatsnew/index.html
|
||||
if sys.version_info >= (3, 2):
|
||||
standard_library.extend([
|
||||
'concurrent',
|
||||
])
|
||||
|
||||
if sys.version_info >= (3, 3):
|
||||
standard_library.extend([
|
||||
'ipaddress',
|
||||
'faulthandler',
|
||||
'lzma',
|
||||
'venv',
|
||||
])
|
||||
|
||||
if sys.version_info >= (3, 4):
|
||||
standard_library.extend([
|
||||
'asyncio',
|
||||
'ensurepip',
|
||||
'enum',
|
||||
'pathlib',
|
||||
'selectors',
|
||||
'statistics',
|
||||
'tracemalloc',
|
||||
])
|
||||
|
||||
if sys.version_info >= (3, 5):
|
||||
standard_library.extend([
|
||||
'typing',
|
||||
'zipapp',
|
||||
])
|
||||
|
||||
if sys.version_info >= (3, 6):
|
||||
standard_library.extend([
|
||||
'secrets',
|
||||
])
|
||||
|
||||
if sys.version_info >= (3, 7):
|
||||
standard_library.extend([
|
||||
'contextvars',
|
||||
'dataclasses',
|
||||
])
|
||||
|
||||
# 'macpath' module has been removed from Python 3.8
|
||||
if sys.version_info > (3, 7):
|
||||
standard_library.remove('macpath')
|
||||
|
||||
# Remove tkinter and Easter eggs
|
||||
excluded_modules = [
|
||||
'antigravity',
|
||||
'this',
|
||||
'turtledemo',
|
||||
]
|
||||
|
||||
def check_missing_modules(expected_modules):
|
||||
missing = []
|
||||
for module in expected_modules:
|
||||
print('Try to import module ', module)
|
||||
try:
|
||||
importlib.import_module(module)
|
||||
except:
|
||||
missing.append(module)
|
||||
return missing
|
||||
|
||||
missing_modules = check_missing_modules(x for x in standard_library if x not in excluded_modules)
|
||||
if missing_modules:
|
||||
print('The following modules are missing:')
|
||||
for module in missing_modules:
|
||||
print(' ', module)
|
||||
exit(1)
|
||||
11
tests/sources/simple-test.py
Normal file
11
tests/sources/simple-test.py
Normal file
@ -0,0 +1,11 @@
|
||||
import sys
|
||||
print(sys.version)
|
||||
print(sys.prefix)
|
||||
|
||||
# Python program to find the factorial of a number
|
||||
num = 65
|
||||
factorial = 1
|
||||
print("Find the factorial of ", num)
|
||||
for i in range(1, num + 1):
|
||||
factorial = factorial*i
|
||||
print("The factorial of ", num, " is ", factorial)
|
||||
Reference in New Issue
Block a user